MLS-C01 Question 385
Select 3You are a data scientist at a retail company that has deployed a machine learning model to predict customer churn. The model is hosted on Amazon SageMaker and predictions are served in real time. While monitoring the model's performance, you notice that the model's accuracy has dropped over the past month. Which steps can you take to identify and address the performance degradation?
- A
Enable Amazon SageMaker Model Monitor to detect data drift and feature distribution changes.
- B
Retrain the model immediately using the same training dataset to restore accuracy.
- C
Analyze inference requests and outputs to identify changes in the input data distribution.
- D
Collect new labeled data and retrain the model to address concept drift.
- E
Enable Amazon CloudWatch Logs to monitor the latency of the model's predictions.
Show answer and explanation
Correct answers: A, C, D
Explanation
Model performance degradation can occur due to data drift, concept drift, or other issues. To monitor and diagnose these problems, you should use tools like Amazon SageMaker Model Monitor to detect data drift, analyze inference inputs and outputs to identify changes, and collect new labeled data to retrain the model if necessary. Simply retraining the model on the same data or monitoring latency will not address the underlying issues.
- A. Correct.
Enabling Amazon SageMaker Model Monitor allows you to detect data drift and changes in feature distributions, which are common causes of model performance degradation.
- B. Incorrect.
Retraining the model immediately using the same training dataset is not recommended because it does not address the root cause of performance degradation, which could be caused by data drift or concept drift.
- C. Correct.
Analyzing inference requests and outputs helps identify if the input data distribution has changed compared to the training data, a critical step in diagnosing performance issues.
- D. Correct.
Collecting new labeled data and retraining the model can address concept drift, which occurs when the relationship between input features and the target variable changes over time.
- E. Incorrect.
While Amazon CloudWatch Logs can monitor latency, it does not provide insights into the model's accuracy or data drift, so it is not directly relevant for addressing performance degradation.