Databricks Generative AI Engineer Associate Question 271
Select 3You have deployed a generative AI model for summarizing customer support tickets in production using Databricks. After monitoring the model's performance over time, you notice the summaries are becoming less accurate and relevant. Which of the following steps should you take to effectively evaluate and address this issue?
- A
Conduct a drift analysis to determine if the input data distribution has changed over time.
- B
Retrain the model using new data from the production environment to improve its performance.
- C
Monitor latency metrics to ensure that the model is performing within the acceptable time limits.
- D
Evaluate the model's summaries against a new test dataset and update evaluation metrics like ROUGE or BLEU scores.
- E
Switch to a larger, more complex model without evaluating the root cause of the issue.
Show answer and explanation
Correct answers: A, B, D
Explanation
When a generative AI model shows reduced performance in production, it's essential to evaluate the root cause by analyzing data drift, updating evaluation metrics, and retraining the model with relevant data if needed. These steps ensure the model adapts to changes over time while maintaining high performance and relevance.
- A. Correct.
Conducting a drift analysis helps identify if the data distribution in production has shifted, which can cause the model to produce less relevant outputs. This step is crucial for determining if data drift is the root cause.
- B. Correct.
Retraining the model on fresh and relevant data from the production environment can help adapt the model to new patterns and improve its performance.
- C. Incorrect.
While monitoring latency is important for production systems, it does not directly address the accuracy or relevance issues of the model's summaries.
- D. Correct.
Evaluating the model's outputs against an updated test dataset and calculating metrics like ROUGE or BLEU helps quantify the model's performance and identify specific areas for improvement.
- E. Incorrect.
Switching to a larger model without understanding the root cause of the issue is not a best practice. It may increase costs unnecessarily and fail to address the underlying problem.