Databricks Generative AI Engineer Associate Question 292
Select 3You are building a Retrieval-Augmented Generation (RAG) application and want to evaluate the model's performance using MLflow. After running several experiments, you want to compare the model's response accuracy and latency for different configurations. Which steps should you follow to effectively log and evaluate performance metrics in MLflow?
- A
Log response accuracy and latency as metrics in MLflow for each experiment run.
- B
Store the vectorized embeddings of documents used for retrieval as parameters in MLflow.
- C
Log the retrieved documents and the final generated responses as artifacts in MLflow.
- D
Use MLflow to automatically optimize the RAG model's hyperparameters without manual configuration.
- E
Compare the metrics across MLflow runs to identify the best-performing configuration.
Show answer and explanation
Correct answers: A, C, E
Explanation
MLflow is a powerful tool for tracking and managing experiments, and in the context of a RAG application, it is essential to log relevant metrics (like accuracy and latency) for performance evaluation. Additionally, storing artifacts such as retrieved documents and generated responses ensures a comprehensive record of the model's outputs. Comparing metrics across runs allows for informed decision-making to select the best-performing configuration.
- A. Correct.
Logging response accuracy and latency as metrics is critical for evaluating the model's performance and comparing different configurations. MLflow provides built-in functionality to log such metrics.
- B. Incorrect.
While vectorized embeddings are important for the retrieval process in a RAG application, they should not be stored as parameters in MLflow. Parameters are typically used for tracking hyperparameters or other configuration values.
- C. Correct.
Logging retrieved documents and generated responses as artifacts in MLflow helps in auditing and debugging model behavior, ensuring traceability of results.
- D. Incorrect.
MLflow does not automatically optimize hyperparameters for RAG models. It is primarily a tracking and lifecycle management tool, not an optimization framework.
- E. Correct.
Comparing metrics across MLflow runs is a key step in identifying the best model configuration. MLflow's UI makes it easy to analyze and compare metrics.