MLS-C01 Question 370
Select 3You are a machine learning engineer at a retail company that uses Amazon Personalize to recommend products to its users. Your team wants to evaluate two different recommendation models (Model A and Model B) by running an A/B test to determine which one generates higher user engagement. Which steps should you take to properly implement the A/B test using AWS services?
- A
Use Amazon Personalize to create campaigns for both Model A and Model B and assign traffic to each model proportionally.
- B
Split the user base randomly into two groups and ensure each group only receives recommendations from one model.
- C
Deploy Amazon SageMaker Model Monitor to track the performance of Model A and Model B during the test.
- D
Define the evaluation criteria (e.g., click-through rate or purchase rate) and measure the performance of each model over the same time period.
- E
Use Amazon CloudWatch to randomly assign users to Model A or Model B.
Show answer and explanation
Correct answers: A, B, D
Explanation
A/B testing requires setting up separate environments for the models being tested, randomly splitting users into groups, and measuring the performance of each model using predefined metrics. In this scenario, Amazon Personalize is used to deploy campaigns for both models, and random user assignment ensures fairness. Evaluation metrics help determine the winning model. Tools like Amazon SageMaker Model Monitor and Amazon CloudWatch are not designed for A/B testing and do not play a role in this specific use case.
- A. Correct.
Correct. Amazon Personalize allows you to create separate campaigns for different models and allocate traffic to test their performance.
- B. Correct.
Correct. Randomly splitting the user base ensures that the test groups are unbiased and comparable.
- C. Incorrect.
Incorrect. Amazon SageMaker Model Monitor is used for monitoring model drift and quality over time, not for A/B testing.
- D. Correct.
Correct. Defining evaluation criteria and measuring model performance over the same time period is critical for a fair A/B test.
- E. Incorrect.
Incorrect. Amazon CloudWatch is a monitoring and logging service, not a tool for assigning users to A/B test groups.