Databricks Generative AI Engineer Associate Question 284
Select 4You are deploying a large language model (LLM) for a customer support chatbot in a production environment. The primary goal is to ensure that the model provides accurate and relevant responses while maintaining low latency during peak traffic. Which metrics should you monitor to ensure the deployment meets these requirements?
- A
Latency (response time)
- B
Token usage per response
- C
Accuracy of responses based on user feedback
- D
Compute resource utilization (e.g., CPU/GPU usage)
- E
Training loss during fine-tuning
- F
Number of concurrent user sessions
Show answer and explanation
Correct answers: A, C, D, F
Explanation
For this LLM deployment scenario, the key metrics to monitor should focus on latency, accuracy, compute utilization, and scalability (e.g., concurrent user sessions). These metrics ensure that the chatbot provides fast, relevant responses while maintaining stability under load. While metrics like token usage and training loss might be useful in other contexts, they are not directly relevant to monitoring a production system for this use case.
- A. Correct.
Latency (response time) is critical for customer-facing applications where users expect quick responses. High latency can negatively impact the user experience.
- B. Incorrect.
Token usage per response may be useful for cost monitoring but is not directly related to the chatbot's performance or user experience in this deployment scenario.
- C. Correct.
Accuracy of responses based on user feedback ensures that the chatbot is providing relevant and correct answers, which is a core requirement for customer support applications.
- D. Correct.
Compute resource utilization should be monitored to ensure the system can handle the load and scale effectively, especially during peak traffic periods.
- E. Incorrect.
Training loss during fine-tuning is relevant during the model training phase, but it is not a key metric to monitor for a production deployment.
- F. Correct.
Number of concurrent user sessions is important to track the system's scalability and ensure that the deployment can handle the expected workload without crashing or degrading performance.