Databricks Machine Learning Associate Question 188
Single answerYou are working on a dataset in Databricks that contains missing values in a numerical column called age. You decide to impute the missing values using the mean of the column. Which of the following approaches will correctly achieve this in Databricks using PySpark?
- A
Fill missing values in the
agecolumn usingdf.fillna(df.select(mean('age')).first()[0], subset=['age']). - B
Use
df.na.fill({'age': df.select(mean('age')).first()[0]})to fill the missing values in theagecolumn. - C
Replace missing values in the
agecolumn usingdf.fillna(df.select(mean('age')).collect()[0][0], subset=['age']). - D
Calculate the mean value separately, store it in a variable, and then use
df.na.fill({'age': mean_value}).
Show answer and explanation
Correct answer: D
Explanation
When imputing missing values with the mean in PySpark, it is necessary to compute the mean value separately, store it in a variable, and then use that variable to replace missing values using df.na.fill. This ensures the computed value is correctly passed into the method used for imputation.
- A. Incorrect.
Incorrect. This syntax is invalid because
fillnadoes not directly accept a scalar value derived from a computation likedf.select(mean('age')).first()[0]. - B. Incorrect.
Incorrect. While
df.na.fillis a valid method for replacing missing values, the syntax here is incorrect because it incorrectly passes the mean computation directly without storing it in a variable first. - C. Incorrect.
Incorrect. This syntax attempts to use
fillna, but it uses a collection object (collect()[0][0]) improperly within the operation. - D. Correct.
Correct. This approach correctly calculates the mean value, stores it in a variable (e.g.,
mean_value), and uses it withdf.na.fill({'age': mean_value})to impute the missing values in theagecolumn.