Databricks Machine Learning Associate Question 187
Single answerYou are working with a dataset on Databricks that contains missing values in a numeric column 'age'. You decide to impute the missing values using the mean of the column. Which of the following steps should you take to achieve this using PySpark?
- A
Use the
fillnamethod on the DataFrame and provide the mean value of the 'age' column. - B
Use the
fillnamethod on the DataFrame and provide the median value of the 'age' column. - C
Calculate the mean of the 'age' column, then use the
fillnamethod with the calculated mean. - D
Use the Databricks AutoML tool to automatically impute missing values with the mean.
Show answer and explanation
Correct answer: C
Explanation
To impute missing values in a numeric column with the mean using PySpark, you need to first calculate the mean of the column and then use the fillna method to replace the missing values. This ensures that the imputation is based on the actual mean value of the column, which is consistent with the requirement described in the question.
- A. Incorrect.
The
fillnamethod can be used to fill missing values, but it requires a specific value to be provided. You must first calculate the mean of the 'age' column before usingfillna, which is not done in this step. - B. Incorrect.
The
fillnamethod supports filling missing values, but the question specifies using the mean, not the median. This step does not align with the specified requirement. - C. Correct.
This is the correct approach. You first calculate the mean of the 'age' column and then use the
fillnamethod to impute the missing values with this calculated mean. - D. Incorrect.
While Databricks AutoML can handle missing values, it is not explicitly designed for manual control over imputation with specific values (e.g., mean or median). This option does not meet the requirement of manually imputing with the mean.