Databricks Generative AI Engineer Associate Question 294
Select 3You have deployed a Retrieval-Augmented Generation (RAG) application on Databricks that uses a large language model (LLM) to provide responses based on retrieved documents. To monitor and improve the system's performance, you decide to enable inference logging. Which of the following actions can you perform using inference logs?
- A
Analyze user queries to identify patterns and improve retrieval performance.
- B
Monitor the latency of each LLM response to optimize system efficiency.
- C
Track the accuracy of the LLM by comparing responses to a ground truth dataset.
- D
Identify documents frequently retrieved but rarely used in LLM responses.
- E
Automatically retrain the LLM based on logged user queries.
Show answer and explanation
Correct answers: A, B, D
Explanation
Inference logging in a RAG application provides critical data about user queries, response times, and retrieval patterns. These logs help engineers analyze system performance, optimize query processing, and refine document retrieval strategies. However, they do not inherently include ground truth data for accuracy tracking or direct mechanisms for automatic model retraining.
- A. Correct.
Correct: Inference logs capture user queries, which can be analyzed to identify patterns and improve the retrieval system's relevancy.
- B. Correct.
Correct: Inference logs often include timing information, allowing you to monitor and optimize response latency.
- C. Incorrect.
Incorrect: Inference logs do not inherently include a ground truth dataset for accuracy comparison; this must be set up separately.
- D. Correct.
Correct: By analyzing inference logs, you can identify documents that are retrieved but not used, helping fine-tune the retrieval strategy.
- E. Incorrect.
Incorrect: Inference logs provide valuable insights but do not directly support automatic LLM retraining without additional workflows.