MLS-C01 Question 204
Single answerYou are building a deep learning model using AWS SageMaker, and the model is not converging during training. Upon reviewing the architecture, you find that the learning rate is set to 0.001, the activation function in the hidden layers is ReLU, and the network consists of multiple dense layers with a large number of nodes. What is the most likely reason for the lack of convergence?
- A
The learning rate is too high, leading to unstable updates.
- B
The ReLU activation function could be causing vanishing gradients.
- C
The learning rate is too low, resulting in slow learning.
- D
The large number of nodes in dense layers is causing overfitting.
Show answer and explanation
Correct answer: C
Explanation
The most likely reason for the lack of convergence is the low learning rate (0.001), which could cause the model to take very small steps during optimization, leading to slow or no progress. Adjusting the learning rate to an appropriate value can help the model converge during training. The other options either do not explain the issue or describe unrelated problems.
- A. Incorrect.
The learning rate being too high would typically cause instability during training, but in this case, the learning rate is relatively small (0.001).
- B. Incorrect.
ReLU activation functions do not cause vanishing gradients; they are designed to mitigate this issue compared to older activation functions like sigmoid or tanh.
- C. Correct.
A low learning rate of 0.001 can result in the model learning too slowly, which is a common cause of non-convergence in deep learning models.
- D. Incorrect.
While a large number of nodes in dense layers could lead to overfitting, it does not directly explain the lack of convergence during training.