Google Professional Machine Learning Engineer Question 317
Select 3Google Cloud PlatformYou are deploying a new version of a recommendation model in production and want to evaluate its performance compared to the current model using A/B testing. The system serves recommendations to users in real time. Which of the following actions are necessary to correctly implement the A/B test and ensure reliable results?
- A
Split users into two groups randomly and consistently assign them to either the current model or the new model.
- B
Measure key performance metrics, such as click-through rate (CTR) and conversion rate, for both models over the same period.
- C
Use the new model for all traffic initially and switch back to the current model if metrics degrade.
- D
Ensure that both models are exposed to the same distribution of input data and user conditions.
- E
Deploy the new model to production immediately for 100% of traffic before analyzing A/B testing results.
Show answer and explanation
Correct answers: A, B, D
Explanation
A/B testing involves splitting traffic between two versions of a model and comparing performance metrics under the same conditions. Randomly and consistently assigning users ensures a fair distribution, while measuring metrics like CTR and ensuring comparable data input avoid biases. Actions like deploying to 100% of traffic or skipping controlled testing contradict the principles of A/B testing.
- A. Correct.
Splitting users randomly and consistently ensures that each user's experience is tied to only one model, avoiding cross-contamination and ensuring reliable A/B test results.
- B. Correct.
Measuring key performance metrics for both models during the same time period ensures that the comparison is fair and not influenced by time-based factors.
- C. Incorrect.
Using the new model for all traffic initially contradicts the goal of A/B testing, which requires a controlled comparison using only a subset of traffic for the new model.
- D. Correct.
Ensuring that both models receive the same distribution of input data and user conditions eliminates biases in data and ensures the test results are comparable.
- E. Incorrect.
Deploying the new model to 100% of traffic before analyzing A/B testing results would bypass the testing phase entirely, defeating the purpose of A/B testing.