Databricks Machine Learning Professional Question 278
Select 3You have trained an updated machine learning model on newly collected data and want to test whether it performs better than the existing model on more recent data. You decide to evaluate both models using a test dataset containing recent data samples. Which of the following steps should you take to ensure a fair comparison and reliable conclusions?
- A
Ensure that the test dataset is independent and not used during the training of either model.
- B
Compare the performance of both models using identical evaluation metrics, such as accuracy or F1 score.
- C
Randomly split the test dataset into smaller subsets to test each model separately to avoid overlap.
- D
Test both models on the exact same test dataset and compare the results.
- E
Use a different test dataset for each model to account for their training differences.
Show answer and explanation
Correct answers: A, B, D
Explanation
To test whether the updated model performs better on more recent data, it is essential to use a single, independent test dataset that was not used during training. Both models should be evaluated on this dataset using the same evaluation metrics to ensure a fair and unbiased comparison. Comparing results under these conditions allows for reliable conclusions about the updated model's performance improvements.
- A. Correct.
Ensuring the test dataset is independent and not used during training is critical to prevent data leakage and ensure an unbiased evaluation of model performance.
- B. Correct.
Using identical evaluation metrics ensures that the comparison between the models is fair and meaningful.
- C. Incorrect.
Randomly splitting the test dataset into smaller subsets is unnecessary and can reduce the statistical power of the evaluation. Both models should be tested on the same complete dataset.
- D. Correct.
Testing both models on the exact same test dataset ensures that the comparison is directly comparable and avoids introducing bias.
- E. Incorrect.
Using a different test dataset for each model is incorrect because any differences in the datasets could introduce confounding variables, making the comparison invalid.