MLS-C01 Question 245
Single answerYou are building a machine learning model to predict customer churn using Amazon SageMaker. During the evaluation phase, you notice that the model's performance varies significantly across different training and testing splits. To address this, you decide to implement cross-validation. Which of the following cross-validation approaches is most appropriate for ensuring consistent evaluation of your model's performance?
- A
Use K-Fold Cross-Validation, where the dataset is divided into K equally sized folds, and the model is trained and tested K times, each time using a different fold as the test set.
- B
Use Leave-One-Out Cross-Validation, where one data point is used as the test set and the rest as the training set, iterating over all data points.
- C
Use Holdout Validation, where the dataset is split into two fixed subsets: one for training and one for testing.
- D
Use Stratified K-Fold Cross-Validation, which ensures each fold preserves the same proportion of classes as the original dataset during splitting.
Show answer and explanation
Correct answer: D
Explanation
Stratified K-Fold Cross-Validation is the best choice for consistent evaluation because it accounts for class imbalance by preserving the class distribution across folds. This is particularly important for imbalanced datasets, such as customer churn, where one class (e.g., churned customers) may be underrepresented. Other methods, such as standard K-Fold or Leave-One-Out, do not explicitly handle class imbalance, while Holdout Validation does not provide the benefits of cross-validation.
- A. Incorrect.
K-Fold Cross-Validation is a valid technique for cross-validation, but it does not account for class imbalance in classification problems, which can lead to biased evaluation metrics.
- B. Incorrect.
Leave-One-Out Cross-Validation is computationally expensive for large datasets and does not address class imbalance, making it less practical for this scenario.
- C. Incorrect.
Holdout Validation is not a cross-validation technique and does not provide multiple evaluations, leading to less reliable performance estimates.
- D. Correct.
Stratified K-Fold Cross-Validation is the most appropriate approach in this scenario, as it ensures each fold preserves the same proportion of classes as the original dataset, addressing class imbalance and providing consistent evaluation metrics.