Databricks Machine Learning Associate Question 184
Select 3You are working on a machine learning project in Databricks where your dataset contains several numerical columns with missing values. You decide to impute the missing values using the mean or median. Which of the following steps should you take to ensure proper imputation and avoid errors in your pipeline?
- A
Calculate the mean or median of each numerical column with missing values using the training dataset.
- B
Replace missing values in the entire dataset (training, validation, and test) with the mean or median calculated from the training dataset.
- C
Calculate and apply the mean or median separately for each data split (training, validation, and test) to ensure consistency within each subset.
- D
Use the imputed values calculated from the test dataset to replace missing values in the training and validation datasets.
- E
Verify that the columns with missing values are numerical, as mean or median imputation is not applicable to categorical data.
Show answer and explanation
Correct answers: A, B, E
Explanation
When imputing missing values with the mean or median in a machine learning pipeline, it is critical to calculate the imputation values using only the training dataset to avoid data leakage. These calculated values should then be applied consistently across the entire dataset (training, validation, and test). Additionally, mean or median imputation is only applicable to numerical columns, so it is essential to verify the data type before proceeding.
- A. Correct.
Correct: The mean or median should always be calculated from the training dataset to prevent data leakage. Using the training dataset ensures that the imputation strategy is not influenced by information from the validation or test datasets.
- B. Correct.
Correct: It is important to use the training dataset's mean or median to replace missing values across all splits (training, validation, and test) to avoid data leakage and ensure consistency.
- C. Incorrect.
Incorrect: Calculating and applying the mean or median separately for each split introduces inconsistencies and risks data leakage, as the test set should not influence the imputation strategy.
- D. Incorrect.
Incorrect: Using the test dataset to calculate imputed values is a form of data leakage, which can lead to overly optimistic evaluation results and invalidate the model's generalizability.
- E. Correct.
Correct: Mean or median imputation is only suitable for numerical columns. For categorical data, other strategies like mode imputation or encoding should be used.