Databricks Data Engineer Associate Question 165
Single answerYou are working with a DataFrame df in Databricks that contains a nested column user with the fields id and name. How can you extract the id field from the user column using dot syntax?
- A
df.select(df.user.id)
- B
df.select('user.id')
- C
df.select(df['user'].id)
- D
df.select(df.user['id'])
Show answer and explanation
Correct answer: A
Explanation
In PySpark, to access nested fields within a column, you can use dot syntax. The correct syntax to extract the id field from the user column is df.user.id. This is the most concise and readable approach to handle nested data fields.
- A. Correct.
This is the correct way to extract a nested field using dot syntax in PySpark. The
user.idfield is accessed usingdf.user.id. - B. Incorrect.
This syntax is incorrect because when using a string inside
select, it does not automatically recognize nested fields. You need to usecol('user.id')or dot syntax directly. - C. Incorrect.
This syntax is invalid because
df['user']does not have an attribute.idin PySpark. - D. Incorrect.
This syntax is also incorrect because
df.user['id']is not supported. You would usedf.user.idfor nested fields.