Databricks Generative AI Engineer Associate Question 290
Select 3You are developing a Retrieval-Augmented Generation (RAG) application to provide accurate, context-aware answers to user queries. To evaluate the performance of your generative model within MLflow, which of the following approaches would be most appropriate?
- A
Log precision, recall, and F1-score as custom metrics in MLflow for generated answers.
- B
Track the latency of the model's response and log it as a custom metric in MLflow.
- C
Log BLEU or ROUGE scores to measure the quality of generated text against ground truth.
- D
Use MLflow to compare embeddings of retrieved documents with embeddings of user queries to evaluate relevance.
- E
Store user feedback (e.g., thumbs up/down) as a parameter within MLflow for performance evaluation.
Show answer and explanation
Correct answers: A, C, D
Explanation
When evaluating the performance of a RAG application in MLflow, it is essential to measure both the relevance of retrieved documents and the quality of generated text. Logging precision, recall, and F1-score provides insight into overall system accuracy, while BLEU or ROUGE scores assess the quality of generated responses. Additionally, comparing embeddings evaluates the retrieval step, ensuring the system retrieves relevant documents. While latency and user feedback are useful metrics, they are not directly tied to the evaluation of the model's generative or retrieval performance.
- A. Correct.
Logging precision, recall, and F1-score provides critical information about the accuracy of generated answers compared to ground truth, which is essential for evaluating model performance in a RAG application.
- B. Incorrect.
While tracking latency is important for assessing model efficiency, it is not directly related to evaluating model performance or accuracy in a RAG application.
- C. Correct.
Metrics like BLEU or ROUGE are standard for evaluating text generation tasks and are useful for assessing how closely the generated text matches the ground truth, making them relevant for a RAG application.
- D. Correct.
Comparing embeddings of retrieved documents with user query embeddings helps measure the relevance of the retrieval step, which is a key component of the overall RAG application performance.
- E. Incorrect.
While storing user feedback is valuable for iterative improvement, it is not a direct way to evaluate model performance within MLflow.