Google Professional Machine Learning Engineer Question 273
Select 3Google Cloud PlatformYou are training a machine learning model using TensorFlow on Vertex AI, but the training job fails repeatedly. The error logs indicate an 'Out of Memory' (OOM) error on the GPU. What steps should you take to troubleshoot and resolve this issue?
- A
Reduce the batch size used during training.
- B
Switch the training job to use a TPU instead of a GPU.
- C
Enable gradient checkpointing to optimize memory usage.
- D
Use preemptible VMs to lower costs and retry the training job.
- E
Increase the GPU memory by selecting a machine with a more powerful GPU.
Show answer and explanation
Correct answers: A, C, E
Explanation
Out of Memory (OOM) errors on GPUs are common when the model, data, or batch size exceeds the GPU's available memory. Reducing the batch size, enabling gradient checkpointing, or selecting a machine with a more powerful GPU are effective ways to troubleshoot and resolve this issue. Switching to TPUs may not directly resolve the problem without additional changes, and using preemptible VMs does not address the memory limitation.
- A. Correct.
Reducing the batch size lowers the memory footprint during training, which can help prevent OOM errors.
- B. Incorrect.
Switching to a TPU may not solve the issue directly as TPUs are different hardware accelerators and may require code modifications. This is not a direct solution to resolving GPU OOM errors.
- C. Correct.
Gradient checkpointing reduces memory usage during backpropagation by storing intermediate states and recomputing them when necessary, which can help address OOM errors.
- D. Incorrect.
Using preemptible VMs focuses on cost optimization and does not address the root cause of the OOM error.
- E. Correct.
Increasing the GPU memory by selecting a machine with a more powerful GPU can help handle larger memory requirements and resolve OOM errors.