Databricks Machine Learning Professional Question 217
Select 3You have deployed a machine learning model to predict customer churn, and it has been running in production for the last three months. Recently, you observed a significant drop in model performance metrics such as accuracy and F1-score. Which of the following steps would be appropriate to investigate and address the issue?
- A
Analyze the feature distributions in the current dataset compared to the training dataset to check for data drift.
- B
Retrain the model immediately with the most recent production data without further investigation.
- C
Monitor prediction confidence levels for recent predictions to identify anomalies.
- D
Check the frequency and distribution of prediction labels to identify potential label drift.
- E
Increase the size of the training dataset by combining it with historical production data without analyzing the impact.
Show answer and explanation
Correct answers: A, C, D
Explanation
To address model performance degradation in production, it is critical to first investigate potential causes such as data drift, label drift, or anomalies in prediction confidence. These steps help ensure that appropriate and informed actions are taken to improve the model. Simply retraining or adding more data without understanding the root cause can lead to suboptimal results or amplify existing issues.
- A. Correct.
Analyzing feature distributions helps identify data drift, which occurs when the statistical properties of input features in production deviate from those in the training dataset. This is a common root cause of performance degradation in deployed models.
- B. Incorrect.
Retraining the model without investigating the root cause of performance issues can be counterproductive. It may lead to overfitting or reinforce the impact of data quality issues in production.
- C. Correct.
Monitoring prediction confidence levels can reveal anomalies, such as sudden increases in low-confidence predictions, which may indicate data drift or issues with the model's generalization.
- D. Correct.
Checking the frequency and distribution of prediction labels helps identify label drift, where the target variable distribution in production differs from the training data, potentially affecting model performance.
- E. Incorrect.
Increasing the training dataset size indiscriminately risks introducing noisy or unrepresentative data into the model. It is important to diagnose the root cause of performance issues before modifying the training data.