MLA-C01 Question 214
Select 3You are tasked with building an ML model for predicting customer churn using Amazon SageMaker. Before deploying the model, you need to establish a performance baseline to evaluate its effectiveness. Which of the following methods can be used to create a meaningful performance baseline?
- A
Use a simple heuristic model, such as predicting churn for all users with a high subscription cancellation rate, and measure its performance.
- B
Train and evaluate a basic machine learning model, such as logistic regression, and use its metrics as a baseline.
- C
Directly deploy the complex model to production and use real-world performance as the baseline.
- D
Use historical data averages, such as the percentage of customers who churned in the past, as a baseline.
- E
Run hyperparameter tuning on advanced models and use the best model's metrics as a baseline.
Show answer and explanation
Correct answers: A, B, D
Explanation
Performance baselines are critical for evaluating whether a new model adds value compared to simpler approaches. Methods like using heuristic models, training basic models, and leveraging historical data averages help establish clear, interpretable reference points. Advanced techniques like hyperparameter tuning or deploying directly to production are not suitable for baseline creation as they are either too complex or lack a comparison framework.
- A. Correct.
Using a simple heuristic model is a valid way to create a baseline. It provides a straightforward approach to compare your complex model's performance against something easily interpretable.
- B. Correct.
Training and evaluating a basic machine learning model like logistic regression is a common method to establish a baseline. It indicates whether a more complex model is worth the effort.
- C. Incorrect.
Directly deploying a complex model to production without a baseline does not provide a meaningful basis for comparison and can lead to unmonitored risks.
- D. Correct.
Using historical data averages can serve as a benchmark for performance because it reflects the natural tendencies of the data without any modeling.
- E. Incorrect.
Running hyperparameter tuning on advanced models is part of optimization, not baseline creation. Baselines are created before optimizing models.