Databricks Generative AI Engineer Associate Question 295
Select 3You have deployed a Retrieval-Augmented Generation (RAG) application using Databricks and want to monitor its performance over time. To achieve this, you decide to use inference logging. Which of the following practices are essential for effectively assessing the performance of the deployed RAG application using inference logs?
- A
Log both the retrieved documents and the generated responses for each query.
- B
Monitor latency metrics to identify whether the RAG pipeline meets performance requirements.
- C
Store inference logs in local storage to ensure faster access for debugging.
- D
Analyze user feedback along with logged responses to evaluate accuracy and relevance.
- E
Enable inference logging only during system failures to reduce storage costs.
Show answer and explanation
Correct answers: A, B, D
Explanation
To effectively assess the performance of a deployed RAG application, it is important to log key components of the pipeline (e.g., retrieved documents and generated responses), monitor system metrics such as latency, and incorporate user feedback to evaluate accuracy and relevance. These practices collectively ensure comprehensive monitoring and improvement of the system over time.
- A. Correct.
Logging both the retrieved documents and generated responses is crucial for assessing whether the retrieval step is providing relevant context and if the generation step is producing accurate and coherent outputs.
- B. Correct.
Monitoring latency metrics is essential to ensure the RAG application meets performance requirements and provides a good user experience.
- C. Incorrect.
Storing inference logs in local storage is not recommended for scalable systems, as cloud or distributed storage options are generally more suitable for handling large-scale data and enabling collaborative debugging.
- D. Correct.
Analyzing user feedback along with logged responses helps in evaluating the accuracy and relevance of the generated output, which is vital for improving the application's performance.
- E. Incorrect.
Enabling inference logging only during system failures undermines continuous performance monitoring and debugging, which are essential for long-term system improvement.