MLS-C01 Question 381
Select 3A retail company has deployed a machine learning model to forecast product demand. After monitoring the model's performance for several weeks, they notice a significant drop in its predictive accuracy. They suspect that the data distribution in production has shifted compared to the training data. Which actions should they take to detect and mitigate this issue?
- A
Enable Amazon SageMaker Model Monitor to track data drift and set up alerts for deviations in input features.
- B
Retrain the model using the original training dataset to ensure consistency in predictions.
- C
Use Amazon CloudWatch metrics to monitor latency and throughput of the model's endpoint.
- D
Perform a statistical analysis on the production data to compare feature distributions with the training data.
- E
Ingest production data into an Amazon S3 bucket and train a new model using both the training data and recent production data.
Show answer and explanation
Correct answers: A, D, E
Explanation
A drop in a model's performance can often be attributed to data drift, where the production data distribution differs from the training data. To address this, you need to detect the drift using tools like Amazon SageMaker Model Monitor and statistical analysis, and then mitigate it by retraining the model with updated data that reflects current production patterns. Simply retraining on the original dataset or monitoring operational metrics like latency won't resolve the root cause of data drift.
- A. Correct.
Correct. Amazon SageMaker Model Monitor can detect data drift by analyzing the input features and comparing them to the training data distribution. Setting up alerts helps detect anomalies in real-time.
- B. Incorrect.
Incorrect. Retraining the model on the original training dataset will not address the issue of data drift, as the training data may no longer reflect the patterns in production data.
- C. Incorrect.
Incorrect. Monitoring latency and throughput in Amazon CloudWatch is important for operational metrics but does not directly address drops in predictive accuracy caused by data drift.
- D. Correct.
Correct. Statistical analysis on the feature distributions can help identify whether data drift has occurred, providing insights into which features have changed in production.
- E. Correct.
Correct. Incorporating recent production data into the training pipeline ensures the model is updated to reflect the current data distribution, mitigating the effects of data drift.