Databricks Generative AI Engineer Associate Question 82
Select 3You are building a retrieval-augmented generation (RAG) system using Databricks for a customer support chatbot. To evaluate the retrieval performance of the system, you decide to use a combination of metrics. Which of the following metrics are suitable for assessing retrieval performance in this context?
- A
Precision@K
- B
Recall@K
- C
BLEU score
- D
Mean Reciprocal Rank (MRR)
- E
Root Mean Square Error (RMSE)
Show answer and explanation
Correct answers: A, B, D
Explanation
To evaluate retrieval performance in a retrieval-augmented generation system, precision@K, recall@K, and mean reciprocal rank (MRR) are standard and effective metrics. These metrics assess the system's ability to retrieve relevant documents and rank them appropriately. BLEU and RMSE are not suitable as they are designed for other types of tasks such as text generation evaluation and regression analysis, respectively.
- A. Correct.
Precision@K is a relevant metric for retrieval performance as it measures the fraction of relevant documents among the top K retrieved results.
- B. Correct.
Recall@K is appropriate for retrieval performance as it measures the fraction of relevant documents retrieved out of all relevant documents in the dataset.
- C. Incorrect.
BLEU score is not suitable for evaluating retrieval performance as it is specifically designed for evaluating the quality of generated text, such as machine translations, by comparing them to reference texts.
- D. Correct.
Mean Reciprocal Rank (MRR) is a valid metric for retrieval performance as it evaluates the rank of the first relevant document in the retrieved results, making it highly relevant for ranking-based systems.
- E. Incorrect.
Root Mean Square Error (RMSE) is not applicable for retrieval performance as it measures the differences between predicted and actual values, which is typically used for regression tasks.