Databricks Generative AI Engineer Associate Question 278
Single answerYou are tasked with selecting an LLM for a customer support chatbot that needs to handle up to 50,000 queries per day with a response latency of less than 2 seconds. The chatbot’s primary requirements are high accuracy in understanding user intent and maintaining response quality for domain-specific queries. You have the following evaluation metrics for three LLM candidates:
Candidate A: 7B parameters, latency 1.8 seconds, domain-specific accuracy 85% Candidate B: 13B parameters, latency 2.5 seconds, domain-specific accuracy 91% Candidate C: 175B parameters, latency 3.2 seconds, domain-specific accuracy 93%
Considering the requirements, which LLM candidate is the most suitable for the chatbot?
- A
Candidate A
- B
Candidate B
- C
Candidate C
- D
None of the above
Show answer and explanation
Correct answer: A
Explanation
The most suitable LLM should meet the latency requirement of less than 2 seconds while providing reasonable domain-specific accuracy for user intent understanding. Candidate A balances these requirements effectively, making it the best choice for the given scenario. Candidates B and C offer higher accuracy but fail to meet the latency constraint, which is critical for the chatbot's performance.
- A. Correct.
Candidate A meets both the latency requirement (1.8 seconds, which is below the 2-second threshold) and provides a reasonable domain-specific accuracy (85%). While not the highest in accuracy, it balances efficiency and quality within the given constraints.
- B. Incorrect.
Candidate B exceeds the latency requirement (2.5 seconds, above the threshold), making it unsuitable despite higher domain-specific accuracy (91%).
- C. Incorrect.
Candidate C also exceeds the latency requirement (3.2 seconds, above the threshold) and is therefore not suitable, even though it has the highest domain-specific accuracy (93%).
- D. Incorrect.
This option is incorrect because Candidate A meets the requirements, so 'None of the above' is not valid.