Databricks Machine Learning Professional Question 218
Select 3You have deployed a machine learning model to predict customer churn, and it is serving predictions in real-time. During a routine audit, you observe that the accuracy of the model has dropped significantly compared to the initial deployment. Which of the following actions should you take to effectively monitor and address the issue?
- A
Implement data drift monitoring to track changes in input feature distributions over time.
- B
Retrain the model periodically without analyzing the root cause of accuracy drop.
- C
Set up an alerting system to notify you when the model's prediction accuracy falls below a predefined threshold.
- D
Monitor model confidence scores to identify patterns of low-confidence predictions.
- E
Manually label a sample of recent data points and calculate performance metrics on this sample.
Show answer and explanation
Correct answers: A, C, E
Explanation
To address a significant drop in model accuracy, it is critical to monitor for data drift, set up alerts for performance degradation, and evaluate the model on fresh, labeled data. These actions help in diagnosing the root cause of the issue and ensuring that the model remains reliable. Simply retraining the model without analysis or relying solely on confidence scores does not provide a comprehensive solution.
- A. Correct.
Implementing data drift monitoring is a proactive approach to identify changes in input feature distributions that could impact model performance. This is crucial in understanding why the accuracy may have dropped.
- B. Incorrect.
Retraining the model periodically without investigating the root cause is not a good practice. This approach may temporarily improve performance but does not address the underlying issue, such as data drift or concept drift.
- C. Correct.
Setting up an alerting system allows you to detect performance degradation in real-time and take timely actions to mitigate the impact on business outcomes.
- D. Incorrect.
Monitoring model confidence scores can help identify specific predictions that are uncertain, but it does not directly address the overall performance degradation.
- E. Correct.
Manually labeling a sample of recent data points and evaluating performance metrics on this sample helps validate how well the model is performing on current data and aids in diagnosing the issue.