Databricks Generative AI Engineer Associate Question 170
Single answerYou are training multiple generative AI models to generate customer support responses. The task requires high-quality, coherent responses with minimal errors, and you evaluate the models using BLEU, ROUGE, and perplexity. The results for three models are as follows:
Model A: BLEU = 0.65, ROUGE = 0.70, Perplexity = 25 Model B: BLEU = 0.60, ROUGE = 0.72, Perplexity = 18 Model C: BLEU = 0.50, ROUGE = 0.55, Perplexity = 12
Which model should you select for this task based on the given metrics?
- A
Model A
- B
Model B
- C
Model C
- D
None of the models meet the requirements
Show answer and explanation
Correct answer: A
Explanation
In generative AI tasks, BLEU and ROUGE scores are commonly used to measure the quality and coherence of model outputs, while perplexity indicates how well the model predicts sequential data. For tasks requiring high-quality and coherent responses, BLEU and ROUGE scores are prioritized. Model A has the best quality metrics and is therefore the most suitable choice.
- A. Correct.
Model A has the highest BLEU and ROUGE scores, which indicate better quality and coherence of generated responses. Although its perplexity is higher than Model B's, the quality metrics take precedence for this task.
- B. Incorrect.
Model B has a lower BLEU and ROUGE score compared to Model A, meaning its generated responses are less aligned with reference answers. While its perplexity is lower, the focus of the task is on quality and coherence.
- C. Incorrect.
Model C has the lowest BLEU and ROUGE scores, and while its perplexity is the lowest, this indicates less complexity rather than higher quality. It is not suitable for the task.
- D. Incorrect.
The models meet the requirements for evaluation, so this option is incorrect.