MLA-C01 Question 14
Select 3You are developing a machine learning model to predict customer churn for a subscription-based business. You decide to use Amazon SageMaker for the task. As part of the model training process, you need to ensure that the data is properly split into training, validation, and test sets to evaluate model performance effectively. Which of the following practices should you follow when splitting the data?
- A
Ensure that the test set is completely unseen during both training and validation phases.
- B
Randomly split the data into training, validation, and test sets without considering data distribution.
- C
Use stratified sampling to maintain the same class proportions in all subsets if the dataset is imbalanced.
- D
Include the test set during hyperparameter tuning to improve the model's performance.
- E
Shuffle the data before splitting to avoid order bias if the data is sequentially arranged.
Show answer and explanation
Correct answers: A, C, E
Explanation
To ensure reliable model evaluation, it is critical to keep the test set unseen, use stratified sampling for imbalanced datasets, and shuffle the data to avoid order bias. These best practices help maintain data integrity and prevent biased performance metrics, which are essential for building robust machine learning models.
- A. Correct.
Correct. The test set should always remain unseen to provide an unbiased evaluation of the model's performance on new data.
- B. Incorrect.
Incorrect. Randomly splitting data without considering distribution can result in subsets that don't reflect the overall dataset, especially for imbalanced data.
- C. Correct.
Correct. Stratified sampling ensures that class distributions in imbalanced datasets are preserved across training, validation, and test sets, leading to better model evaluation.
- D. Incorrect.
Incorrect. Including the test set during hyperparameter tuning can lead to overfitting and an inflated evaluation metric, as the model indirectly learns from the test data.
- E. Correct.
Correct. Shuffling data before splitting is important if the data is sequential (e.g., time-series or sorted by category) to avoid introducing biases in the splits.