Databricks Generative AI Engineer Associate Question 276
Single answerYou are designing a generative AI application for summarizing legal documents. The application must prioritize accuracy and factuality in summaries, while maintaining minimal latency for real-time usability. Based on the following evaluation metrics of three pre-trained LLMs, which model is the most appropriate choice?
Model A:
- BLEU Score: 85
- Latency: 500ms
- Factuality Score: 90
Model B:
- BLEU Score: 80
- Latency: 300ms
- Factuality Score: 95
Model C:
- BLEU Score: 78
- Latency: 200ms
- Factuality Score: 92
- A
Model A
- B
Model B
- C
Model C
- D
None of the models meet the requirements
Show answer and explanation
Correct answer: B
Explanation
Model B is the best choice because it balances the key requirements of accuracy (BLEU score), factuality, and latency. While Model A has higher accuracy, its latency is too high for real-time use. Model C has the lowest latency but underperforms in both accuracy and factuality. Model B provides the optimal trade-off for the given summarization task.
- A. Incorrect.
Model A has the highest BLEU score (accuracy) but suffers from high latency (500ms), which does not meet the minimal latency requirement for real-time usability.
- B. Correct.
Model B achieves a good balance of factuality (95), BLEU score (80), and latency (300ms). While its BLEU score is slightly lower than Model A, its superior factuality and lower latency make it the most appropriate choice.
- C. Incorrect.
Model C has the lowest BLEU score (78) and factuality (92), although it has the lowest latency (200ms). However, its lower accuracy and factuality disqualify it for the use case.
- D. Incorrect.
This option is incorrect because Model B meets the requirements with a good balance of factuality, accuracy, and latency.