Databricks Generative AI Engineer Associate Question 110
Select 3You are tasked with evaluating responses generated by a large language model (LLM) for a customer service application. During your assessment, you notice issues like hallucinations (false information), offensive language, and irrelevant content. Which of the following actions should you take to qualitatively assess and improve the model's responses?
- A
Create a rubric to evaluate responses based on accuracy, relevance, and tone.
- B
Ignore responses with hallucinations since they are rare and not impactful.
- C
Perform a manual review of a diverse set of generated responses to identify recurring issues.
- D
Rely solely on automated metrics like BLEU or ROUGE scores to evaluate the quality of the responses.
- E
Collect user feedback and analyze it to identify patterns and areas of improvement.
Show answer and explanation
Correct answers: A, C, E
Explanation
Qualitatively assessing responses involves both structured evaluations (like rubrics) and hands-on testing (manual review). Combining this with user feedback ensures a comprehensive understanding of the model's strengths and weaknesses. Ignoring critical issues like hallucinations or relying solely on automated metrics can lead to unsafe or low-quality outcomes.
- A. Correct.
Creating a rubric helps establish a systematic and consistent way to evaluate the model's responses based on critical dimensions like accuracy, relevance, and tone.
- B. Incorrect.
Ignoring hallucinations is not advisable, as they can significantly harm the user experience and trust in the system.
- C. Correct.
Performing a manual review allows you to identify nuanced issues, such as offensive or irrelevant content, that automated tools might miss.
- D. Incorrect.
While automated metrics like BLEU or ROUGE are helpful for initial evaluation, they do not capture all aspects of response quality, such as safety and tone.
- E. Correct.
User feedback is essential for understanding real-world performance and for identifying patterns that can guide further improvements.