MLS-C01 Question 236
Single answerA data science team at a retail company has deployed a new machine learning model to recommend products to users. The team wants to compare the new model's performance against the existing model in a live environment to determine if it improves user engagement. Which approach should the team use to perform this evaluation?
- A
Conduct offline evaluation using the historical dataset to compare the two models' performance metrics.
- B
Perform online evaluation by setting up an A/B testing framework and splitting traffic between the two models.
- C
Use cross-validation with the historical dataset to evaluate both models' accuracy and select the better one.
- D
Deploy the new model in production for all users and compare its performance over time to the old model's historical performance.
Show answer and explanation
Correct answer: B
Explanation
A/B testing is a standard method for online evaluation where live traffic is split between multiple models to measure their performance in real-world conditions. This approach ensures the new model's effectiveness is assessed based on real-time user interactions. Offline evaluation methods like cross-validation or historical analysis are better suited for initial testing but do not account for live production dynamics.
- A. Incorrect.
Offline evaluation using a historical dataset is not suitable for comparing model performance in a live environment. It does not account for real-time user behavior or dynamic factors in the production environment.
- B. Correct.
A/B testing is the correct approach for online evaluation. It allows the team to split live traffic between the two models and directly compare their performance on real user interactions.
- C. Incorrect.
Cross-validation using a historical dataset is an offline evaluation method that is insufficient to determine performance in a live production environment.
- D. Incorrect.
Deploying the new model to all users without comparison may lead to negative business outcomes if the model underperforms. It also does not allow for a direct, controlled comparison with the old model.