NCP-AII Question 13
Select 4You are managing an AI training cluster that consists of multiple NVIDIA GPUs. One of your training jobs is underperforming, and the system logs show intermittent power fluctuations and thermal throttling warnings for one of the GPUs. What steps should you take to identify and address the faulty components?
- A
Check the GPU utilization and thermal metrics using tools like nvidia-smi.
- B
Inspect the system's power supply for voltage irregularities and ensure it meets the required wattage specifications.
- C
Replace the entire GPU cluster with new GPUs to avoid further issues.
- D
Examine the PCIe connection and reseat the GPU to ensure proper installation.
- E
Run diagnostic tests provided by NVIDIA tools to identify hardware issues with the GPU.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To address the issue, a step-by-step diagnostic approach is necessary. Tools like nvidia-smi provide real-time GPU metrics that can reveal thermal or performance-related issues. Power supply inspections ensure the GPU is receiving adequate power. Checking the physical connection (PCIe) and running NVIDIA diagnostic tools help confirm whether the GPU or its associated components are at fault. Replacing the entire GPU cluster is unnecessary and not aligned with a targeted troubleshooting approach.
- A. Correct.
Checking GPU utilization and thermal metrics using tools like nvidia-smi can help identify if the specific GPU is overheating or experiencing thermal throttling.
- B. Correct.
Inspecting the power supply for voltage irregularities ensures that the GPU is receiving adequate and stable power, which is critical for performance.
- C. Incorrect.
Replacing the entire GPU cluster is unnecessary and costly without pinpointing the exact issue. This is not a practical or efficient approach.
- D. Correct.
Examining the PCIe connection and reseating the GPU can resolve issues caused by improper installation or loose connections.
- E. Correct.
Running diagnostic tests using NVIDIA tools can help pinpoint hardware faults, such as power delivery issues or memory errors, in the GPU.