Databricks Generative AI Engineer Associate Question 289
Select 4You are tasked with evaluating a Retrieval-Augmented Generation (RAG) application deployed on Databricks. The application uses a language model to generate answers to user queries using external knowledge retrieved from a vector database. How can you use MLflow to evaluate the performance of the RAG application?
- A
Log the retrieval accuracy (e.g., precision@k) as an MLflow metric to measure how often relevant documents are retrieved.
- B
Log the model's generated responses as MLflow artifacts for later qualitative evaluation.
- C
Use MLflow's built-in confusion matrix functionality to evaluate the classification accuracy of the RAG application.
- D
Log BLEU or ROUGE scores as MLflow metrics to assess the quality of the generated responses.
- E
Track latency of the retrieval and generation steps as MLflow metrics to monitor system performance.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Evaluating a RAG application involves analyzing both the retrieval and generation components. MLflow provides tools to track metrics, artifacts, and system performance. Logging retrieval accuracy, generation quality scores (such as BLEU or ROUGE), and system latency are key steps in quantifying the performance of the RAG application. Additionally, qualitative evaluation of responses can be facilitated by storing outputs as artifacts. Confusion matrices, however, are not applicable in this context as RAG applications are not classification problems.
- A. Correct.
Logging retrieval accuracy as an MLflow metric is crucial for understanding how well the retrieval component of the RAG application is functioning.
- B. Correct.
Logging generated responses as MLflow artifacts enables qualitative and manual evaluation of the model's outputs, which is an important part of assessing performance.
- C. Incorrect.
MLflow does not provide built-in functionality for confusion matrices, as this is typically used for classification tasks, not RAG applications.
- D. Correct.
Logging BLEU or ROUGE scores as MLflow metrics is appropriate for evaluating the quality of the natural language generation in a RAG application.
- E. Correct.
Tracking latency as a metric in MLflow provides valuable insights into the system's performance and user experience, as latency impacts usability.