Databricks Machine Learning Professional Question 236
Select 3You have deployed a machine learning model to production that predicts customer churn. After a few weeks, you notice that the model's performance has significantly degraded. To investigate, you suspect data drift might be the cause. Which steps should you take to detect and monitor for data drift in this scenario?
- A
Compare the statistical distribution of input features in the training dataset with the production dataset.
- B
Use a feature importance tool to identify which features are contributing most to the model's predictions in production.
- C
Implement a drift detection test that monitors the real-time distribution of predicted probabilities over time.
- D
Manually retrain the model on new production data and compare its accuracy with the existing model.
- E
Set up alerts to trigger when the input features in production deviate significantly from the training data.
Show answer and explanation
Correct answers: A, C, E
Explanation
Data drift occurs when the statistical properties of input features or the target variable change over time, impacting model performance. To address this, you should compare training and production data distributions, monitor predicted probabilities for concept drift, and set up alerts for significant deviations. These steps ensure proactive detection and monitoring of data drift in production environments.
- A. Correct.
Comparing the statistical distribution of input features between training and production datasets is a standard method for identifying data drift, as it helps detect significant changes in the input data.
- B. Incorrect.
Using a feature importance tool is useful for understanding model interpretability but does not directly address the detection or monitoring of data drift.
- C. Correct.
Monitoring the distribution of predicted probabilities over time helps detect concept drift, where the relationship between input features and the target variable changes.
- D. Incorrect.
Manually retraining the model is a reactive step and does not help in detecting or monitoring data drift proactively.
- E. Correct.
Setting up alerts for deviations in input features is a proactive way to monitor data drift and ensure timely interventions when distributions change.