MLS-C01 Question 206
Select 2You are training a deep learning model on AWS to classify images into different categories. During the initial training, you notice that the model is not learning effectively, and the loss is not decreasing significantly after several epochs. Upon reviewing the neural network architecture, you find that you are using many layers with ReLU activation functions and a learning rate of 0.0001. What changes could you make to improve the model's training performance?
- A
Increase the learning rate to a moderately higher value, such as 0.01.
- B
Replace the ReLU activation functions with sigmoid activation functions.
- C
Add batch normalization layers between the dense layers.
- D
Reduce the depth of the neural network by using fewer layers.
- E
Switch the optimizer to stochastic gradient descent (SGD) without momentum.
Show answer and explanation
Correct answers: A, C
Explanation
The model's poor performance during training is likely due to a combination of a learning rate that is too low and a lack of mechanisms to stabilize training. Increasing the learning rate to a reasonable value allows the model to make larger updates to weights, improving convergence speed. Adding batch normalization helps normalize layer inputs and mitigate issues like internal covariate shift, which can improve training stability and efficiency. The other options either do not address the issue or could introduce new problems, such as vanishing gradients or slower optimization.
- A. Correct.
Increasing the learning rate to a moderately higher value like 0.01 can help the model converge faster. A very low learning rate of 0.0001 can lead to slow training progress.
- B. Incorrect.
Replacing the ReLU activation functions with sigmoid is not advisable as sigmoid functions can lead to the vanishing gradient problem, especially in deep networks.
- C. Correct.
Adding batch normalization layers can stabilize and accelerate training by normalizing the inputs to each layer, which can help improve convergence.
- D. Incorrect.
Reducing the depth of the neural network might not be necessary unless there is clear evidence of overfitting or architectural inefficiency. In this case, the issue is related to training dynamics, not overfitting.
- E. Incorrect.
Switching to SGD without momentum might slow down the optimization process. Optimizers like Adam are generally preferred for deep learning tasks as they adapt the learning rate and include momentum.