MLS-C01 Question 384
Select 4An e-commerce company has deployed a recommendation system using Amazon SageMaker. Over the past week, the system's performance has degraded, with a significant drop in click-through rates (CTR). As the machine learning engineer, you are tasked with detecting the root cause and mitigating the performance drop. Which of the following approaches should you use?
- A
Analyze the input data for changes in data distribution using Amazon SageMaker Data Wrangler or Amazon S3 logs.
- B
Enable Amazon CloudWatch metrics to monitor system performance and set up alarms for anomalies.
- C
Update the SageMaker model endpoint to a higher instance type to ensure more compute and memory resources are available.
- D
Retrain the model using recent data and compare the evaluation metrics with the current model’s performance.
- E
Use Amazon SageMaker Model Monitor to detect data quality and concept drift in real time.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Drops in ML model performance can often be attributed to data-related issues such as changes in data distribution or concept drift. Steps like analyzing input data, monitoring system metrics, retraining the model with recent data, and leveraging tools like SageMaker Model Monitor are essential for identifying and mitigating such issues. Upgrading the instance type alone does not address the root cause of the problem.
- A. Correct.
Analyzing input data for changes in distribution is crucial, as changes in data patterns can lead to model performance drops. Tools like Amazon SageMaker Data Wrangler help identify such changes.
- B. Correct.
Enabling Amazon CloudWatch metrics and alarms allows you to monitor system performance metrics, such as latency and error rates, which can signal performance issues.
- C. Incorrect.
Simply upgrading the instance type does not address the root cause of a performance drop, which is more likely due to data or model-related issues.
- D. Correct.
Retraining the model with recent data helps address potential issues caused by outdated training data or concept drift, improving the model’s relevance to current data patterns.
- E. Correct.
Amazon SageMaker Model Monitor is specifically designed to detect data quality issues and concept drift, enabling you to take corrective action in real time.