MLA-C01 Question 215
Select 3You are developing a machine learning model for a financial fraud detection system using Amazon SageMaker. To evaluate the model's performance, you need to establish a performance baseline. Which of the following methods can be used to create an effective performance baseline for this model?
- A
Use historical data to calculate metrics such as precision, recall, and F1-score for existing manual or rule-based systems.
- B
Deploy the model without evaluation and observe its performance directly in production.
- C
Split the dataset into training, validation, and test sets, and use the test set to measure model performance.
- D
Use synthetic data generated by tools like Amazon SageMaker Ground Truth to simulate performance metrics.
- E
Compare model performance against a random classifier to assess its relative improvement.
Show answer and explanation
Correct answers: A, C, E
Explanation
Establishing a performance baseline is critical for evaluating and improving machine learning models. Methods such as using historical data, splitting datasets appropriately for evaluation, and using comparisons with a random classifier are standard practices to create a reliable baseline. These approaches ensure the new model's performance can be measured effectively and improvements can be quantified.
- A. Correct.
Using historical data to calculate metrics provides a benchmark for comparing the new model's performance against existing approaches, helping to establish a baseline.
- B. Incorrect.
Deploying the model without evaluation skips the critical step of offline testing and validation, which is essential for establishing a baseline in a controlled environment.
- C. Correct.
Splitting the dataset into training, validation, and test sets ensures a reliable evaluation of the model's performance on unseen data, which is a standard method for creating baselines.
- D. Incorrect.
While synthetic data can be useful for certain tasks, it may not accurately represent real-world conditions, making it less reliable for establishing performance baselines.
- E. Correct.
Comparing model performance against a random classifier helps to quantify relative improvement, providing a meaningful benchmark for evaluation.