MLS-C01 Question 210
Select 3You are training a gradient-boosted tree model using Amazon SageMaker's built-in XGBoost algorithm for a binary classification task. During hyperparameter tuning, you observe that the model is overfitting the training data. Which of the following hyperparameter configurations could help reduce overfitting?
- A
Decrease the maximum depth of the trees (
max_depth) - B
Increase the number of trees (
num_round) - C
Increase the minimum child weight (
min_child_weight) - D
Decrease the learning rate (
eta) - E
Decrease the subsample ratio (
subsample)
Show answer and explanation
Correct answers: A, C, E
Explanation
To reduce overfitting in tree-based models like gradient-boosted trees, you can adjust hyperparameters to limit model complexity and introduce randomness. Decreasing the maximum depth (max_depth) simplifies the structure of the trees. Increasing the minimum child weight (min_child_weight) ensures that splits occur only when sufficient data supports them, making the model less prone to overfit. Decreasing the subsample ratio (subsample) introduces randomness, which can help the model generalize better to unseen data. Conversely, increasing the number of trees or reducing the learning rate may exacerbate overfitting or have minimal impact on overfitting directly.
- A. Correct.
Decreasing the maximum depth of the trees limits the complexity of the model and helps reduce overfitting by preventing it from capturing too much noise in the training data.
- B. Incorrect.
Increasing the number of trees (
num_round) can lead to overfitting if not carefully controlled, as more trees can capture noise from the training data, making this option incorrect. - C. Correct.
Increasing the minimum child weight forces the model to group more data points into each leaf node, which results in simpler trees and helps reduce overfitting.
- D. Incorrect.
While decreasing the learning rate (
eta) slows down the learning process and can improve generalization over many iterations, it does not directly address overfitting and is less effective than the other options. - E. Correct.
Decreasing the subsample ratio (
subsample) limits the amount of data used to build each tree, which introduces randomness and reduces overfitting.