Databricks Generative AI Engineer Associate Question 81
Select 3You are working on a retrieval-augmented generation (RAG) system where a vector store is used to retrieve relevant documents to answer user queries. To evaluate the retrieval performance of your system, you decide to use specific metrics. Which of the following metrics would be most appropriate to measure retrieval performance?
- A
Mean Reciprocal Rank (MRR)
- B
Precision@K
- C
BLEU score
- D
Recall@K
- E
Word Error Rate (WER)
Show answer and explanation
Correct answers: A, B, D
Explanation
The evaluation of retrieval performance in a system like retrieval-augmented generation (RAG) focuses on how effectively relevant documents are retrieved. Metrics such as Mean Reciprocal Rank (MRR), Precision@K, and Recall@K are specifically designed to measure different aspects of retrieval performance. BLEU and WER, on the other hand, are unrelated to retrieval and are used for evaluating text generation or speech-to-text systems.
- A. Correct.
Mean Reciprocal Rank (MRR) is a common metric for evaluating retrieval systems. It measures the rank of the first relevant document in the retrieved list, making it highly relevant for evaluating retrieval performance.
- B. Correct.
Precision@K assesses the proportion of relevant documents in the top K results, making it a suitable metric for evaluating retrieval performance in systems like RAG.
- C. Incorrect.
BLEU score is used for evaluating the quality of machine-generated text against a reference text. It is not relevant for measuring retrieval performance in a RAG system.
- D. Correct.
Recall@K measures the proportion of all relevant documents retrieved within the top K results. It is an important metric for evaluating the comprehensiveness of a retrieval system.
- E. Incorrect.
Word Error Rate (WER) is used to evaluate the accuracy of speech-to-text systems. It is not applicable for evaluating retrieval performance.