MLS-C01 Question 162
Select 3You are training a deep learning model on AWS SageMaker, and you observe that the training loss decreases initially but then starts oscillating without converging. Which of the following optimization techniques could help address this issue?
- A
Reduce the learning rate of the optimizer
- B
Switch to a different loss function
- C
Use a learning rate scheduler to adjust the learning rate dynamically
- D
Increase the batch size used for training
- E
Switch from SGD (Stochastic Gradient Descent) to Adam optimizer
Show answer and explanation
Correct answers: A, C, E
Explanation
When training loss oscillates without converging, it often indicates issues with the learning rate or the optimization algorithm. Reducing the learning rate or using a learning rate scheduler can stabilize the training process. Additionally, switching to an optimizer like Adam, which adapts the learning rate for each parameter, can also help address this issue. Changing the loss function or increasing the batch size may not directly resolve oscillations caused by such optimization challenges.
- A. Correct.
Reducing the learning rate can help stabilize training and prevent oscillations, particularly when the learning rate is too high.
- B. Incorrect.
Changing the loss function is unlikely to directly address oscillation issues, as the problem is more related to the optimization process rather than the specific loss function used.
- C. Correct.
Using a learning rate scheduler can dynamically adjust the learning rate during training, helping the model gradually converge by reducing the rate as training progresses.
- D. Incorrect.
Increasing the batch size may improve the stability of gradient updates, but it is not the most effective solution for addressing oscillations in training.
- E. Correct.
Switching to the Adam optimizer, which uses adaptive learning rates, can help mitigate oscillations and improve convergence in training.