Databricks Machine Learning Professional Question 237
Select 3You are deploying a machine learning model for predicting customer churn, and you want to implement drift monitoring to ensure the model remains accurate over time. Which of the following scenarios would most likely indicate data drift and require further investigation?
- A
The distribution of the input features in the production data significantly differs from the training data distribution.
- B
The model's accuracy on recent production data is consistent with the accuracy observed during training.
- C
A sharp increase in the frequency of certain categorical feature values not seen during training.
- D
The model's predictions in production have a similar distribution to the predictions on the training data.
- E
The target variable's distribution in production data shows significant deviation from the training data distribution.
Show answer and explanation
Correct answers: A, C, E
Explanation
Drift monitoring involves identifying changes in the data or model outputs that can degrade the performance of a machine learning system. Input feature drift, the appearance of new feature values, and target variable drift are key signs of potential issues that require investigation. However, consistent accuracy or similar output distributions are not necessarily indicative of drift.
- A. Correct.
Significant differences in the distribution of input features between training and production data often indicate data drift, which may affect model performance.
- B. Incorrect.
Consistent accuracy between training and production suggests the model is generalizing well, and there is no clear evidence of drift in this case.
- C. Correct.
A sharp increase in unseen categorical feature values is a strong indicator of data drift, as the model may not have been trained to handle these new values.
- D. Incorrect.
Similar distributions of predictions in training and production suggest no significant drift in the model's output behavior.
- E. Correct.
A significant deviation in the target variable's distribution between training and production data indicates concept drift, as the relationship between inputs and outputs may have changed.