MLS-C01 Question 371
Single answerYou are a machine learning specialist at an e-commerce company that wants to test two versions of a product recommendation model to determine which one leads to higher customer engagement. The models are deployed behind an Amazon API Gateway using AWS Lambda functions. You need to implement A/B testing to compare the two versions (Model A and Model B) while ensuring fair traffic distribution and tracking user behavior. Which approach should you take?
- A
Configure two separate AWS Lambda functions for Model A and Model B, and use Amazon CloudWatch Logs to track the traffic distribution.
- B
Deploy both models on AWS Lambda, configure an Amazon API Gateway stage variable to route traffic, and use Amazon CloudWatch Metrics and custom logging to measure the results.
- C
Create a single AWS Lambda function that randomly selects between Model A and Model B, and use AWS X-Ray for traffic tracking and analysis.
- D
Use AWS Step Functions to orchestrate the traffic distribution between Model A and Model B and analyze results using Amazon S3 logs.
Show answer and explanation
Correct answer: B
Explanation
A/B testing involves splitting traffic between two variants (in this case, Model A and Model B) in a controlled manner and monitoring the outcomes to determine which performs better. Using Amazon API Gateway stage variables allows you to route traffic effectively between the models, and Amazon CloudWatch Metrics with custom logging provides a robust mechanism to track user engagement and model performance. This makes Option 2 the best choice.
- A. Incorrect.
This option is incorrect because while it enables running both Lambda functions, it does not provide an effective mechanism to control traffic distribution or measure the results in a structured way.
- B. Correct.
This option is correct because using API Gateway stage variables allows you to define routing rules for traffic distribution between Model A and Model B. Additionally, Amazon CloudWatch Metrics and custom logging support monitoring and analyzing the performance of each model effectively.
- C. Incorrect.
This option is incorrect because while random selection can distribute traffic, it does not offer a reliable or configurable mechanism to ensure fair traffic distribution or detailed tracking for A/B testing.
- D. Incorrect.
This option is incorrect because AWS Step Functions is designed for workflow orchestration and not for traffic routing or A/B testing. Additionally, it is not the optimal tool for tracking model performance and engagement metrics.