MLA-C01 Question 371
Select 3You are developing a machine learning model to predict customer churn for a subscription-based service. The dataset is stored in an Amazon S3 bucket, and you decide to use Amazon SageMaker for training and deployment. To ensure your model performs well, you want to split the data into training, validation, and test datasets. Which of the following practices should you follow to correctly handle the data split?
- A
Randomly split the dataset into training, validation, and test sets while ensuring class distribution is maintained across sets.
- B
Use the same dataset for training, validation, and testing to maximize the amount of data available for model training.
- C
Ensure that no data leakage occurs by keeping the test set completely separate and only using it after model training and validation are complete.
- D
Manually assign samples to the training, validation, and test sets to ensure the best possible performance of the model.
- E
Use stratified sampling if the dataset is imbalanced to ensure equal representation of classes across the splits.
Show answer and explanation
Correct answers: A, C, E
Explanation
Properly splitting data into training, validation, and test sets is a critical step in machine learning. Random splits with class distribution maintenance and stratified sampling for imbalanced datasets ensure the subsets reflect the original data. Additionally, keeping the test set separate avoids data leakage and provides a reliable evaluation of the model's performance on unseen data.
- A. Correct.
Randomly splitting the data while maintaining the class distribution ensures that all subsets are representative of the dataset, which is crucial for building a robust model.
- B. Incorrect.
Using the same dataset for training, validation, and testing can lead to overfitting and unreliable performance metrics, as the model is evaluated on the same data it was trained on.
- C. Correct.
Keeping the test set completely separate prevents data leakage and ensures that the model's performance is evaluated on unseen data, which is essential for an unbiased evaluation.
- D. Incorrect.
Manually assigning samples is not a scalable or reproducible practice and can introduce bias, leading to unreliable model performance.
- E. Correct.
Stratified sampling is a best practice when dealing with imbalanced datasets, as it ensures that all subsets have similar class distributions, improving the model's generalization ability.