Databricks Data Engineer Associate Question 166
Single answerYou are working with a nested JSON dataset in Databricks that contains the following structure:
{ "user": { "id": 101, "name": "John Doe", "preferences": { "notifications": true, "theme": "dark" } } }
You need to extract the 'theme' value from the 'preferences' field. Which of the following code snippets correctly extracts this field using dot syntax in PySpark?
- A
df.select('user.preferences.theme')
- B
df.select('user.preferences["theme"]')
- C
df.select(col('user.preferences.theme'))
- D
df.select(col('user.preferences')['theme'])
Show answer and explanation
Correct answer: A
Explanation
Dot syntax in PySpark allows you to directly access nested fields in a DataFrame. The correct way to select the 'theme' field under 'preferences' is by writing 'user.preferences.theme' within the 'select' function. Other options either misuse the dot syntax or attempt to combine it with unsupported methods like square brackets.
- A. Correct.
This is the correct syntax to use dot notation for extracting a nested field in PySpark.
- B. Incorrect.
This syntax is incorrect because the brackets with a string key are not supported in dot syntax when directly selecting fields.
- C. Incorrect.
This syntax is incorrect because 'col' does not support dot syntax directly for nested field extraction.
- D. Incorrect.
This syntax is incorrect because you cannot use square brackets with 'col' to directly extract nested fields.