Google Professional Machine Learning Engineer Question 510
Select 3Google Cloud PlatformYou are deploying a machine learning model to predict customer churn for a subscription-based service. To ensure the model remains effective in production, you decide to monitor its performance over time. Which of the following strategies are appropriate for monitoring the model's performance against baselines and detecting potential issues?
- A
Monitor the model's prediction accuracy against the baseline model established before deployment.
- B
Track drift in input feature distributions over time to identify changes in the data.
- C
Compare the current model's predictions with a simpler model, such as a rule-based system.
- D
Rely solely on monitoring the model's training loss to ensure model performance.
- E
Periodically re-calibrate the baseline model based on the latest production data.
Show answer and explanation
Correct answers: A, B, C
Explanation
Effective model monitoring involves tracking its performance against the baseline established during development, identifying data drift, and using simpler models for comparison to detect unexpected behavior. Solely relying on training loss or arbitrarily re-calibrating baselines without evaluation does not provide a reliable way to monitor model effectiveness in production.
- A. Correct.
Monitoring the model's prediction accuracy against the baseline model helps ensure that the deployed model continues to perform as expected compared to the initial benchmark.
- B. Correct.
Tracking drift in input feature distributions is crucial as it helps identify changes in the data that can lead to performance degradation over time.
- C. Correct.
Comparing the deployed model's predictions with a simpler model, such as a rule-based system, can provide additional context and highlight when the deployed model might be underperforming.
- D. Incorrect.
Monitoring training loss is insufficient in production since it only reflects performance on the training dataset and does not account for issues such as data drift or changes in the real-world environment.
- E. Incorrect.
Re-calibrating the baseline model without careful evaluation can lead to biases and undermines the purpose of having a fixed benchmark for performance monitoring.