Databricks Generative AI Engineer Associate Question 296
Select 4You have deployed a Retrieval-Augmented Generation (RAG) application on Databricks, and users have started interacting with it. To assess the performance of the application and identify areas for improvement, the team decides to use inference logging. Which of the following steps should you take to effectively use inference logging for performance assessment?
- A
Enable logging of both queries and retrieved documents during inference.
- B
Analyze latency metrics for each user query to detect potential bottlenecks.
- C
Exclude failed queries from inference logs to avoid skewing the performance metrics.
- D
Log user feedback on generated responses alongside query and retrieval data.
- E
Perform periodic reviews of inference logs to identify trends in irrelevant retrievals.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Inference logging for a deployed RAG application involves capturing and analyzing data related to queries, retrieved documents, response generation, and user feedback. This comprehensive logging approach helps identify performance issues, such as latency bottlenecks or irrelevant retrievals, and provides actionable insights for optimizing the application. Excluding failed queries would omit critical data and hinder the ability to improve the system.
- A. Correct.
Correct. Logging queries and retrieved documents is essential for understanding how the application retrieves and generates responses, which is critical for performance assessment.
- B. Correct.
Correct. Latency analysis helps identify if there are performance bottlenecks in query processing or response generation, which is key for optimization.
- C. Incorrect.
Incorrect. Excluding failed queries would result in incomplete data and prevent proper assessment of failure patterns, which are important for improving system robustness.
- D. Correct.
Correct. Logging user feedback provides valuable context for assessing the quality of generated responses and can guide fine-tuning of the RAG pipeline.
- E. Correct.
Correct. Periodic log reviews can help identify trends, such as recurring irrelevant retrievals, which can be addressed to improve application performance.