Databricks Machine Learning Professional Question 277
Select 2You have an updated version of a machine learning model that you want to test against a more recent dataset to determine if it performs better than the current model. You decide to evaluate both models on a holdout set from this recent dataset. Which of the following steps should you take to ensure a statistically sound comparison?
- A
Split the recent dataset into training, validation, and test sets to evaluate both models
- B
Use the same random seed to ensure both models are evaluated on the same test data split
- C
Perform a paired statistical test, such as a paired t-test, to compare the performance metrics of both models
- D
Compare the models using a single metric, such as accuracy, to determine which one performs better
- E
Use cross-validation on the recent dataset to assess both models’ performance
Show answer and explanation
Correct answers: B, C
Explanation
To test whether the updated model performs better than the current model on a new dataset, it is essential to evaluate both models on the same test data split. Using a consistent random seed ensures fair comparison, and applying a paired statistical test, such as a paired t-test, provides evidence of whether the performance difference is statistically significant. These steps ensure a rigorous and unbiased evaluation.
- A. Incorrect.
This is incorrect. Splitting the dataset into training, validation, and test sets is important for model development, but the focus here is on evaluating two trained models. The validation and training splits are not needed for this specific comparison.
- B. Correct.
This is correct. Using the same random seed ensures that both models are evaluated on the exact same test data, avoiding any bias caused by differences in data splits.
- C. Correct.
This is correct. A paired statistical test, such as a paired t-test, is necessary to determine if the observed differences in performance metrics (e.g., accuracy, precision, etc.) are statistically significant.
- D. Incorrect.
This is incorrect. Comparing models using a single metric without additional statistical testing can lead to incorrect conclusions, as it does not account for variance or provide statistical significance.
- E. Incorrect.
This is incorrect. While cross-validation is a robust technique for model evaluation, it is not typically used to directly compare the performance of two pre-trained models on a specific dataset.