Databricks Machine Learning Associate Question 170
Select 3A data scientist is building a predictive model using a dataset that contains missing values in several features. The missing values are replaced using mean imputation. Why is it important to add indicator variables for the missing values?
- A
It allows the model to account for the fact that missing data may carry useful information.
- B
It ensures that imputation does not distort the overall distribution of the data.
- C
It helps the model differentiate between the imputed values and the original values.
- D
It prevents overfitting caused by the imputed values.
- E
It provides a way to analyze whether missingness itself is predictive of the target variable.
Show answer and explanation
Correct answers: A, C, E
Explanation
Adding indicator variables for missing values allows the model to capture patterns related to missingness, which might carry useful information. This can help the model differentiate between imputed and original values, and also analyze whether missingness itself is predictive of the target variable. These benefits make indicator variables an important step when dealing with imputed data.
- A. Correct.
Correct: Missing values might carry meaningful information. Adding an indicator variable allows the model to capture patterns related to whether a value was missing, which may be predictive.
- B. Incorrect.
Incorrect: Imputation methods like mean imputation do not inherently preserve the data's original distribution. Indicator variables do not resolve this issue, but they help model the missingness itself.
- C. Correct.
Correct: Adding indicator variables helps the model distinguish between imputed and original values. Otherwise, the model may treat all values as if they were observed, potentially missing key patterns.
- D. Incorrect.
Incorrect: Adding indicator variables does not prevent overfitting. Overfitting depends on how the model is trained, regularized, or validated.
- E. Correct.
Correct: Indicator variables allow the model to assess whether the absence of data (or missingness mechanism) is predictive of the target variable. This is often valuable in real-world datasets.