AIF-C01 Question 54
Select 3A financial services company is deploying a machine learning model to detect fraudulent transactions. The team wants to ensure the deployed model remains accurate over time. They need to monitor the model’s performance in production and retrain it when necessary due to data drift. Which of the following strategies should the team implement to achieve this goal?
- A
Set up a pipeline to automatically retrain the model at fixed time intervals regardless of data changes.
- B
Use Amazon SageMaker Model Monitor to track data quality and distribution in real-time.
- C
Establish a feedback loop to capture user-validated results and integrate them into training data for retraining.
- D
Manually compare the model's performance metrics every six months to decide if retraining is required.
- E
Implement automated alerts for significant deviations in model performance metrics.
Show answer and explanation
Correct answers: B, C, E
Explanation
To manage model performance in production and address data drift, it is critical to implement a combination of automated monitoring tools, feedback loops, and alert systems. Amazon SageMaker Model Monitor provides real-time insights into data quality, while feedback loops and automated alerts ensure the model remains accurate and reliable over time. This strategy ensures production readiness and minimizes technical debt associated with model degradation.
- A. Incorrect.
Automatically retraining the model at fixed intervals is inefficient and may not address the actual occurrence of data drift or performance degradation.
- B. Correct.
Amazon SageMaker Model Monitor can continuously analyze and detect changes in data quality and distribution, which is essential for identifying data drift in production.
- C. Correct.
Establishing a feedback loop to capture validated results helps collect real-world data for retraining, ensuring the model adapts to evolving patterns.
- D. Incorrect.
Manually comparing metrics every six months is not scalable or responsive enough for production-grade systems where real-time monitoring is critical.
- E. Correct.
Setting up automated alerts for deviations in model performance ensures timely notifications when the model's accuracy or other key metrics degrade.