Databricks Generative AI Engineer Associate Question 287
Select 4You are deploying a large language model (LLM) for a customer support chatbot that must provide accurate and timely responses. Which metrics should you monitor to ensure the LLM deployment meets the business requirements?
- A
Latency of responses
- B
Token usage per conversation
- C
Word count of responses
- D
Response accuracy
- E
User satisfaction score
Show answer and explanation
Correct answers: A, B, D, E
Explanation
For an LLM-based customer support chatbot, monitoring metrics like latency, token usage, response accuracy, and user satisfaction ensures that both technical performance and business goals are met. These metrics provide insights into the efficiency, correctness, and user perception of the chatbot. In contrast, metrics like word count do not directly align with the goals of this specific deployment scenario.
- A. Correct.
Latency of responses is crucial to ensure the chatbot delivers timely answers, especially for customer support use cases where delays can negatively impact user experience.
- B. Correct.
Token usage per conversation is important to monitor as it directly impacts cost, especially in scenarios where LLM usage is billed per token.
- C. Incorrect.
Word count of responses is not a key metric for this deployment scenario. While concise responses may be useful, word count alone does not guarantee quality or relevance.
- D. Correct.
Response accuracy is essential to ensure that the chatbot provides correct and useful information to the user, which is critical for maintaining trust and effectiveness in customer support.
- E. Correct.
User satisfaction score is a key business-oriented metric to measure how well the chatbot meets user expectations and improves overall customer experience.