MLS-C01 Question 369
Select 3You are a machine learning engineer at an e-commerce company. Your team wants to test the impact of a new personalized recommendation model on user engagement. To do this, you decide to conduct an A/B test by deploying the new model to 50% of users (Group B) while keeping the current model for the other 50% (Group A). Which of the following steps are necessary to properly design and evaluate this A/B test in AWS?
- A
Randomly assign users to Group A and Group B to ensure unbiased results
- B
Use Amazon CloudWatch to monitor real-time recommendations made by the new model
- C
Define a clear success metric, such as click-through rate (CTR), to measure user engagement
- D
Use Amazon SageMaker Model Monitor to ensure the input data distribution remains consistent between the groups
- E
Run the test for a statistically significant amount of time to account for any daily or seasonal variations
Show answer and explanation
Correct answers: A, C, E
Explanation
To conduct a proper A/B test, it is crucial to ensure that groups are unbiased through random assignment, success metrics are clearly defined to evaluate outcomes, and the test duration is sufficient to account for statistical significance. Additional tools like Amazon SageMaker Model Monitor or CloudWatch can support operational aspects but are not directly tied to the core A/B testing framework.
- A. Correct.
Correct. Randomly assigning users to A and B ensures that there is no inherent bias in the test groups, which could otherwise skew the results.
- B. Incorrect.
Incorrect. While monitoring the recommendations in real-time can provide insights into system performance, this is not directly related to A/B test design or evaluation.
- C. Correct.
Correct. Defining a clear success metric, like CTR, is essential to measure the outcome and effectiveness of the new model in a quantifiable way.
- D. Incorrect.
Incorrect. While Amazon SageMaker Model Monitor is useful for monitoring data drift and model quality, ensuring consistent data distribution between groups is not typically its role in A/B testing.
- E. Correct.
Correct. Running the test for a statistically significant period ensures that results are not influenced by short-term anomalies or seasonal effects, improving reliability.