MLA-C01 Question 385
Select 3You are designing a machine learning system on AWS to predict customer churn. To ensure effective monitoring of the ML model in production, you decide to follow best practices outlined in the ML Lens of the AWS Well-Architected Framework. Which of the following actions should you take to ensure proper monitoring of the ML model's performance over time?
- A
Implement model performance monitoring by tracking metrics such as accuracy, precision, recall, and F1 score on a regular basis.
- B
Set up Amazon CloudWatch Alarms to monitor infrastructure metrics such as CPU utilization and memory usage for the ML model endpoint.
- C
Enable Amazon SageMaker Model Monitor to track data drift and model quality drift in real-time.
- D
Perform a one-time evaluation of the model's accuracy before deploying it to production and use this as the baseline for monitoring.
- E
Log inference requests and responses to Amazon S3 for offline analysis and anomaly detection.
Show answer and explanation
Correct answers: A, C, E
Explanation
Monitoring an ML model in production involves tracking both its performance and the data it processes. Regularly evaluating metrics like accuracy, leveraging tools like Amazon SageMaker Model Monitor, and logging inference data for offline analysis are all critical steps in ensuring model reliability and mitigating issues like data drift or performance degradation. Infrastructure monitoring alone is insufficient, and one-time evaluations do not address the dynamic nature of production environments.
- A. Correct.
Correct: Monitoring model performance metrics is essential to ensure the model behaves as expected in production. Regular tracking of accuracy, precision, recall, F1 score, and other metrics helps identify performance degradation over time.
- B. Incorrect.
Incorrect: While monitoring infrastructure metrics is important for system health, it does not directly address the ML model's performance, such as data drift or prediction quality.
- C. Correct.
Correct: Amazon SageMaker Model Monitor is a purpose-built tool for detecting data drift and model quality drift, which are critical for maintaining model performance in production.
- D. Incorrect.
Incorrect: A one-time evaluation is insufficient for monitoring. Continuous or periodic evaluations are necessary as data distributions and model behavior can change over time.
- E. Correct.
Correct: Logging inference requests and responses allows for detailed offline analysis, including anomaly detection and understanding how the model performs across various data slices.