NCP-AII Question 7
Select 3You are managing an AI training workload on a server equipped with NVIDIA GPUs. During training, the system reports degraded performance and logs indicate intermittent 'GPU unresponsive' errors. What are the possible steps to identify and troubleshoot the hardware fault?
- A
Check the GPU temperature and ensure it is within safe operating limits.
- B
Inspect the server's networking hardware for disconnected cables or faulty ports.
- C
Verify the GPU power connection and ensure it is properly seated.
- D
Run NVIDIA System Management Interface (nvidia-smi) to check for GPU utilization and error logs.
- E
Restart the training workload without investigating further, as the issue may resolve itself.
Show answer and explanation
Correct answers: A, C, D
Explanation
In this scenario, the issue is specific to the GPU ('GPU unresponsive' errors), so the troubleshooting steps should focus on hardware and software diagnostics related to the GPU. Checking the temperature, ensuring proper power connections, and leveraging tools like nvidia-smi are effective methods to identify potential GPU faults. Networking hardware and simply restarting the workload are irrelevant or ineffective in this context.
- A. Correct.
Monitoring the GPU's temperature is critical, as overheating can lead to performance degradation or hardware failure. This is a valid troubleshooting step.
- B. Incorrect.
While networking issues can affect distributed training setups, they are unrelated to the 'GPU unresponsive' error in this scenario.
- C. Correct.
A loose or improperly seated GPU power connection can cause intermittent GPU failures. This step is appropriate for troubleshooting.
- D. Correct.
Running nvidia-smi provides valuable insights into GPU utilization, error logs, and other diagnostics, making this an important step in identifying GPU-related faults.
- E. Incorrect.
Restarting the workload without investigation does not address the root cause and is not a recommended troubleshooting step.