Databricks Generative AI Engineer Associate Question 288
Select 3You are building a Retrieval-Augmented Generation (RAG) application using a large language model (LLM) and a vector database for document retrieval. You want to evaluate the model's performance using MLflow to compare different LLMs and retrieval strategies. Which of the following steps are necessary to effectively evaluate and track model performance in this scenario?
- A
Log the LLM's outputs, retrieval results, and evaluation metrics as MLflow artifacts for each experiment.
- B
Manually record the model's hyperparameters and evaluation results in a spreadsheet for comparison.
- C
Use MLflow's registered models feature to version and compare the LLMs used in the RAG pipeline.
- D
Log custom evaluation metrics, such as retrieval accuracy and response relevance, using MLflow tracking.
- E
Use MLflow to deploy the RAG pipeline directly without tracking intermediate results.
Show answer and explanation
Correct answers: A, C, D
Explanation
To evaluate the performance of a RAG application using MLflow, you need to log relevant artifacts, track custom metrics, and utilize MLflow's model registry for versioning and comparison. These steps ensure that all aspects of the application, including retrieval accuracy and LLM performance, are properly evaluated and can be improved iteratively. Manual methods or skipping intermediate logging are not recommended as they hinder scalability and reproducibility.
- A. Correct.
Logging the LLM's outputs, retrieval results, and evaluation metrics as MLflow artifacts allows you to track and analyze all relevant data for each experiment, which is essential for evaluating model performance.
- B. Incorrect.
Manually recording data in a spreadsheet is not an efficient or scalable approach, especially when MLflow provides automated tracking and comparison features.
- C. Correct.
MLflow's model registry enables versioning and comparison of different models, helping you track improvements in the RAG pipeline as you test different LLMs or configurations.
- D. Correct.
Custom evaluation metrics, such as retrieval accuracy and response relevance, are highly specific to RAG applications and should be logged in MLflow for thorough performance evaluation.
- E. Incorrect.
MLflow can be used for deployment, but skipping the tracking of intermediate results would make it difficult to evaluate and improve the RAG application.