MLS-C01 Question 383
Select 3A machine learning model deployed on Amazon SageMaker has been serving predictions in production for several months. Recently, you noticed a drop in model performance, as indicated by lower accuracy metrics in your monitoring system. Which actions should you take to detect and mitigate the root cause of the performance drop?
- A
Enable Amazon SageMaker Model Monitor to track data quality issues and compare the current input data distribution against the training dataset.
- B
Retrain the model immediately using the latest production data without investigating the root cause of the performance drop.
- C
Analyze the feature importance of the model to identify if certain features’ distributions have shifted over time.
- D
Use Amazon CloudWatch to check for any infrastructure issues, such as insufficient instance resources or high latency, which might be causing degraded performance.
- E
Perform hyperparameter tuning using Amazon SageMaker Automatic Model Tuning to improve the model's current performance.
Show answer and explanation
Correct answers: A, C, D
Explanation
Drops in model performance can occur due to various reasons, such as input data drift, feature distribution changes, or infrastructure issues. The correct approach involves first diagnosing the root cause using tools like Amazon SageMaker Model Monitor to detect data quality issues, analyzing feature importance to check for feature drift, and using Amazon CloudWatch to identify infrastructure problems. Jumping directly to retraining or hyperparameter tuning without identifying the root cause can lead to inefficient use of time and resources.
- A. Correct.
Correct: Amazon SageMaker Model Monitor can detect distribution shifts in input data or violations of data quality constraints, which are common causes of performance drops.
- B. Incorrect.
Incorrect: Retraining the model without investigating the root cause could lead to wasted resources or further degradation if the issue lies in data quality or infrastructure. Root cause analysis is essential.
- C. Correct.
Correct: Analyzing feature importance can help identify if certain features that were significant during training have shifted in their distribution, which may explain the performance drop.
- D. Correct.
Correct: Infrastructure issues such as high latency, insufficient resources, or other anomalies can impact the overall performance of the deployed model. Amazon CloudWatch is a useful tool for diagnosing these issues.
- E. Incorrect.
Incorrect: Hyperparameter tuning is not the first step to address a performance drop. It is more effective after confirming that data quality and infrastructure are not the root causes.