MLS-C01 Question 197
Select 2You are building a machine learning model for a binary classification problem using Amazon SageMaker. To ensure the model generalizes well to unseen data, you decide to perform cross-validation. Which of the following approaches would correctly implement cross-validation in this scenario?
- A
Split the dataset into k subsets (folds), use k-1 folds for training, and the remaining fold for validation, repeating the process k times and averaging the results.
- B
Randomly split the dataset into training and validation sets once, train the model on the training set, and evaluate on the validation set.
- C
Use SageMaker's built-in k-fold cross-validation functionality to automatically generate folds and perform training and validation.
- D
Manually split the dataset into k folds, train the model k times, each time leaving out one fold for validation, and calculate the average performance metric across all runs.
- E
Perform hyperparameter tuning using SageMaker Automatic Model Tuning (also known as hyperparameter optimization), which inherently uses cross-validation.
Show answer and explanation
Correct answers: A, D
Explanation
Cross-validation is a robust method for evaluating a machine learning model's performance by splitting the dataset into k subsets (or folds). The model is trained on k-1 folds and validated on the remaining fold, iterating this process k times. This ensures that each data point is used for validation exactly once, reducing bias and variance in the evaluation. Options 1 and 4 describe valid implementations of k-fold cross-validation, while the other options either describe different concepts or are incorrect.
- A. Correct.
This is the correct definition and approach for implementing k-fold cross-validation. It ensures that every data point gets a chance to be in both the training and validation sets, reducing the risk of overfitting.
- B. Incorrect.
This describes a simple train-test split, not cross-validation. While it is a valid approach, it does not provide the robustness of cross-validation and is not what the question asks for.
- C. Incorrect.
SageMaker does not have a built-in functionality specifically for k-fold cross-validation. Users need to implement cross-validation manually or through custom scripts.
- D. Correct.
This is a correct way to manually implement k-fold cross-validation. By training the model on k-1 folds and validating on the left-out fold in an iterative manner, you can ensure robust performance estimation.
- E. Incorrect.
SageMaker Automatic Model Tuning (hyperparameter optimization) does not inherently use cross-validation. It evaluates models on a separate validation set during the tuning process, but this is not equivalent to cross-validation.