Databricks Generative AI Engineer Associate Question 270
Select 3You are deploying a generative AI model in Databricks to generate personalized product descriptions for an e-commerce platform. After deployment, you notice that the model occasionally generates inaccurate or irrelevant descriptions. Which of the following steps should you take to evaluate and monitor the model's performance effectively?
- A
Implement a feedback loop to allow users to flag inaccurate descriptions.
- B
Monitor input data drift to detect changes in the incoming data distribution over time.
- C
Use a static evaluation metric, such as BLEU score, without periodic re-evaluation.
- D
Establish an automated pipeline for retraining the model on new labeled data.
- E
Ignore monitoring since the model performed well during initial offline evaluation.
Show answer and explanation
Correct answers: A, B, D
Explanation
To ensure the generative AI model remains useful and accurate in production, it is crucial to implement a robust evaluation and monitoring strategy. This includes collecting user feedback, monitoring data drift, and establishing mechanisms for retraining the model. Static evaluation or ignoring monitoring are insufficient approaches for maintaining model quality in dynamic environments.
- A. Correct.
Implementing a feedback loop allows for collecting real-world insights from users, helping you identify inaccuracies and retrain the model as needed. This is an essential part of monitoring a deployed generative AI model.
- B. Correct.
Monitoring input data drift ensures that the model's performance is not degraded due to shifts in the data distribution over time. This is a critical aspect of model monitoring.
- C. Incorrect.
Using a static evaluation metric without periodic re-evaluation is insufficient because the model's performance may degrade over time as data or use cases evolve.
- D. Correct.
Establishing an automated retraining pipeline ensures the model remains up-to-date and can adapt to new data or changes in user behavior. This is a best practice for long-term monitoring and maintenance.
- E. Incorrect.
Ignoring monitoring is not advisable because even well-performing models during offline evaluation may fail in production due to real-world complexities or evolving data patterns.