Databricks Generative AI Engineer Associate Question 172
Single answerYou are tasked with selecting the best generative AI model for a text summarization task. You have experimented with three models and evaluated them on the following metrics: BLEU score, ROUGE-L score, model inference time (lower is better), and perplexity. Based on the results below, which model should you select for the task?
Model A: BLEU = 0.45, ROUGE-L = 0.62, Inference Time = 200ms, Perplexity = 15 Model B: BLEU = 0.48, ROUGE-L = 0.70, Inference Time = 300ms, Perplexity = 14 Model C: BLEU = 0.41, ROUGE-L = 0.65, Inference Time = 250ms, Perplexity = 16
- A
Model A
- B
Model B
- C
Model C
- D
None of the models meet the criteria for text summarization
Show answer and explanation
Correct answer: B
Explanation
When selecting a model for text summarization, BLEU and ROUGE-L are the most relevant metrics for evaluating the quality of the generated summaries. Perplexity is a secondary metric that can indicate language model fluency, while inference time is important but can often be traded off for better quality. Model B outperforms the others in BLEU and ROUGE-L, making it the best choice for this task.
- A. Incorrect.
Model A has lower BLEU and ROUGE-L scores compared to Model B, which indicates it performs worse in terms of language generation quality and summarization accuracy.
- B. Correct.
Model B has the highest BLEU and ROUGE-L scores, which are the most critical metrics for evaluating the quality of text summarization. While its inference time is slightly higher, the tradeoff is acceptable given its superior quality.
- C. Incorrect.
Model C has higher ROUGE-L than Model A but lower BLEU and higher perplexity than Model B. It also has a moderate inference time, but its overall quality does not surpass Model B.
- D. Incorrect.
All three models meet the criteria for text summarization based on standard metrics. This option is incorrect because one model clearly outperforms the others.