MLS-C01 Question 388
Select 3You have deployed a machine learning model using Amazon SageMaker and need to monitor its performance in production to ensure it continues to make accurate predictions. Which of the following actions should you take to monitor the model’s performance over time?
- A
Enable Amazon SageMaker Model Monitor to analyze incoming prediction requests and actual outcomes, and configure it to detect data drift.
- B
Use CloudWatch Metrics to monitor the latency and throughput of the model endpoint.
- C
Set up an endpoint to retrain the model automatically whenever the accuracy drops below a predefined threshold.
- D
Periodically evaluate model predictions against ground truth data to calculate performance metrics, such as accuracy or F1 score.
- E
Enable AWS Config to track changes in the SageMaker model endpoint configuration.
Show answer and explanation
Correct answers: A, B, D
Explanation
Monitoring a deployed machine learning model involves tracking both operational metrics (e.g., latency, throughput) and performance metrics (e.g., accuracy, F1 score). Amazon SageMaker Model Monitor provides automated capabilities to detect changes in data, such as data drift, and CloudWatch Metrics helps track the operational health of the endpoint. Periodic evaluation of the model's predictions against ground truth ensures that the model continues to perform as expected. These combined actions help maintain and monitor the model's performance effectively.
- A. Correct.
Correct: Amazon SageMaker Model Monitor can be used to track data drift, bias, and other changes in the data to ensure the model is working as expected.
- B. Correct.
Correct: CloudWatch Metrics can be used to track operational metrics like latency and throughput, which are important for ensuring the endpoint is functioning properly.
- C. Incorrect.
Incorrect: While retraining is important, setting up an endpoint to automatically retrain a model is not always recommended. Model retraining should involve careful evaluation and validation steps.
- D. Correct.
Correct: Comparing model predictions against ground truth periodically allows you to calculate performance metrics and ensure the model remains reliable over time.
- E. Incorrect.
Incorrect: AWS Config is used for tracking configuration changes in your AWS environment, but it does not directly help in monitoring model performance.