Databricks Generative AI Engineer Associate Question 293
Select 3You are working on a Retrieval-Augmented Generation (RAG) application and want to evaluate the model's performance using MLflow. Which of the following steps are necessary to ensure an effective evaluation of your RAG model's performance in MLflow?
- A
Log the retrieval step accuracy (e.g., precision or recall) as a custom metric in MLflow.
- B
Log the generated answers from the model alongside the associated query as MLflow artifacts.
- C
Record the latency of the retrieval and generation steps as separate metrics in MLflow.
- D
Only log the final accuracy of the RAG pipeline as a single metric in MLflow.
- E
Use MLflow's built-in RAG evaluation functions to automate model scoring.
Show answer and explanation
Correct answers: A, B, C
Explanation
When evaluating a RAG application in MLflow, it is crucial to evaluate both the retrieval and generation components individually. Logging metrics like retrieval accuracy, latency, and generated outputs as artifacts provides a comprehensive view of the model's performance. This approach ensures alignment with real-world requirements and enables iterative improvements to the pipeline.
- A. Correct.
Correct. Logging the retrieval accuracy as a custom metric in MLflow is essential to understand how well the retrieval component of the RAG pipeline is performing.
- B. Correct.
Correct. Logging the generated answers and associated queries as artifacts ensures traceability and allows for qualitative evaluation of the RAG model's outputs.
- C. Correct.
Correct. Logging latency metrics for both retrieval and generation steps is necessary to evaluate the efficiency of the RAG pipeline, which is critical in real-world applications.
- D. Incorrect.
Incorrect. While logging the final accuracy is important, it is insufficient for a comprehensive evaluation of a RAG pipeline. Each component (retrieval and generation) must be evaluated separately.
- E. Incorrect.
Incorrect. MLflow does not provide built-in RAG evaluation functions. Custom metrics, parameters, and artifacts must be logged explicitly to evaluate RAG applications.