Databricks Data Engineer Associate Question 169
Single answerYou are working with a JSON dataset in a Databricks notebook. The dataset contains the following nested structure for each record: { "user": { "id": 123, "details": { "name": "John Doe", "age": 30 } } }. You want to extract the name field from the details object using PySpark. Which of the following code snippets correctly extracts this field?
- A
df.select('user.details.name')
- B
df.select('details.name')
- C
df.select(col('user.details.name'))
- D
df.select(col('user.details').name)
Show answer and explanation
Correct answer: C
Explanation
To access nested fields in a PySpark DataFrame, the dot syntax can be used in combination with the col function. In this case, the correct syntax to extract the name field is col('user.details.name') as it correctly navigates through the nested structure of the dataset.
- A. Incorrect.
This option is incorrect because PySpark requires the use of the
colfunction to access nested fields with dot syntax. - B. Incorrect.
This option is incorrect because the field
details.nameis not at the top level of the dataset. It is nested under theuserfield. - C. Correct.
This is the correct option because the
colfunction supports the dot syntax to access nested fields in the dataset. - D. Incorrect.
This option is incorrect because the
col('user.details')expression retrieves the entiredetailsobject, and.namecannot be directly appended to it.