Databricks Machine Learning Professional Question 279
Select 2You are working on a machine learning project in Databricks where an updated model has been trained using more recent data. The updated model is now being evaluated to determine if it performs better than the current production model on the most recent dataset. Which of the following steps should you take to test if the updated model outperforms the production model?
- A
Split the most recent dataset into training, validation, and test sets, and compare the performance of both models on the test set.
- B
Use a statistical test, such as a paired t-test, to compare the performance metrics (e.g., accuracy, F1-score) of the two models on the most recent dataset.
- C
Deploy the updated model to production and observe real-world performance metrics over time.
- D
Perform cross-validation on the most recent dataset using both models and compare their average performance metrics.
- E
Directly compare the training losses of the updated and production models on their respective training datasets to evaluate performance.
Show answer and explanation
Correct answers: B, D
Explanation
To test whether the updated model performs better on the more recent data, you should compare the performance of both models using a robust evaluation method. Statistical tests, such as a paired t-test, help determine if performance differences are significant. Cross-validation provides a reliable way to evaluate and compare models on the same dataset. These methods ensure that the updated model is rigorously tested before deployment.
- A. Incorrect.
Splitting the most recent dataset into training, validation, and test sets is not necessary when comparing two pre-trained models. This approach introduces unnecessary overhead and does not directly address the comparison of model performance on the same dataset.
- B. Correct.
Using a statistical test, like a paired t-test, is a valid method to determine whether the performance differences between the two models are statistically significant on the same dataset.
- C. Incorrect.
Deploying the updated model to production without prior evaluation is risky and not a recommended practice for comparing model performance. Evaluation should be done offline first.
- D. Correct.
Cross-validation on the most recent dataset helps to robustly compare the performance of both models by computing average performance metrics, making it an appropriate step in the evaluation process.
- E. Incorrect.
Comparing training losses is not an appropriate method to evaluate the performance of models on new data. Training loss only reflects performance on the training dataset and does not indicate generalization to the test or recent data.