Databricks Generative AI Engineer Associate Question 283
Select 3You are deploying a large language model (LLM) in a customer support chatbot application. The primary goal is to ensure the chatbot provides accurate responses to customer queries while maintaining low latency. Which key metrics should you monitor to ensure the performance and reliability of this deployment?
- A
Response accuracy
- B
Latency (response time)
- C
Model training loss
- D
User engagement rate
- E
Token usage per response
Show answer and explanation
Correct answers: A, B, E
Explanation
In this scenario, the key metrics to monitor focus on the LLM's ability to deliver accurate and efficient responses in real time. Response accuracy ensures the chatbot provides relevant and correct answers, latency ensures the responses are delivered promptly, and token usage per response helps manage resource consumption and cost. Metrics like training loss or user engagement rate are less relevant for monitoring the deployment of an already-trained LLM in production.
- A. Correct.
Response accuracy is critical to ensure that the chatbot provides correct answers to user queries, which directly impacts the quality of the customer support experience.
- B. Correct.
Latency (response time) is crucial for user satisfaction, as long delays in generating responses can lead to frustration and a poor user experience.
- C. Incorrect.
Model training loss is not relevant in this deployment scenario because the model has already been trained and is being used in production. Monitoring training loss applies during the training phase, not during deployment.
- D. Incorrect.
User engagement rate is a secondary metric in this context. While it can provide insights into overall system usage, it does not directly measure the LLM's performance or reliability in responding to queries.
- E. Correct.
Token usage per response is important because it can impact the cost efficiency and resource utilization of the deployment. High token usage might indicate inefficiencies in response generation.