Databricks Generative AI Engineer Associate Question 274
Select 3A team has deployed a generative AI model to production using Databricks. They want to ensure that the model is performing well over time and identify any drift in its outputs when compared to expected results. Which of the following actions should they take to effectively evaluate and monitor the model?
- A
Implement automated logging of model predictions and corresponding ground truth data for comparison.
- B
Set up alerting mechanisms for significant deviations in key evaluation metrics such as BLEU score or perplexity.
- C
Manually evaluate the model's outputs every week by sampling random predictions.
- D
Track feature distributions over time to detect data drift in the input data.
- E
Periodically retrain the model without monitoring, assuming that data evolution will occur.
Show answer and explanation
Correct answers: A, B, D
Explanation
Effective evaluation and monitoring of a generative AI model in production requires a combination of automated tools and processes. Logging predictions and ground truth, setting up alerts for metric deviations, and monitoring input data drift are essential practices to identify performance issues and data changes over time. Manual evaluations and blind retraining are either inefficient or risky approaches in this context.
- A. Correct.
Correct: Automated logging of predictions and ground truth data is essential for continuous evaluation and monitoring, allowing the team to track model performance over time.
- B. Correct.
Correct: Setting up alerting mechanisms ensures the team is immediately notified of significant performance deviations, which is critical for early detection of issues in production.
- C. Incorrect.
Incorrect: While manual evaluation can provide insights, it is not scalable or efficient for monitoring a production model over time.
- D. Correct.
Correct: Tracking feature distributions helps detect input data drift, which can significantly impact the performance of a generative AI model.
- E. Incorrect.
Incorrect: Retraining the model without monitoring can lead to suboptimal results if there is no clear understanding of the model's performance or the underlying data issues.