NCP-AII Question 15
Select 2You are managing an AI infrastructure with multiple NVIDIA GPUs in a data center designed to handle high-performance workloads. Recently, one of the models running on the cluster experienced significant performance degradation. Upon investigation, you suspect hardware issues. Which of the following steps are most effective in identifying faulty GPUs or related hardware components?
- A
Use NVIDIA System Management Interface (nvidia-smi) to check for GPU errors and monitor utilization statistics.
- B
Physically inspect all GPUs and power supplies for visible signs of damage or overheating.
- C
Run stress tests with diagnostic tools like NVIDIA Data Center GPU Manager (DCGM) to detect instability or errors.
- D
Reboot the system to clear any temporary software issues and re-run the workload to confirm consistent performance.
- E
Replace all GPUs in the cluster as a precautionary measure.
Show answer and explanation
Correct answers: A, C
Explanation
Diagnosing faulty GPUs or related hardware components requires a systematic approach. Tools like nvidia-smi and DCGM are specifically designed to monitor and stress-test NVIDIA GPUs, providing detailed insights into potential hardware issues. Physical inspection and system rebooting can be supplementary steps but are not as effective or scalable for diagnosing hardware problems in AI infrastructure. Prematurely replacing all GPUs is not a recommended practice without confirmed evidence of widespread failure.
- A. Correct.
Using nvidia-smi allows you to monitor GPU utilization, temperature, and error states, which are critical for identifying potential hardware faults. It is a standard practice for diagnosing GPU-related issues.
- B. Incorrect.
While physically inspecting components can help identify visible damage, it is often impractical for large-scale clusters and does not provide detailed diagnostic data on GPU performance or faults.
- C. Correct.
Running stress tests with tools like DCGM is an effective way to identify hardware instability or errors under load. It complements the data obtained from nvidia-smi.
- D. Incorrect.
Rebooting the system might clear temporary issues, but it is not a direct method for diagnosing hardware faults. It is more of a general troubleshooting step.
- E. Incorrect.
Replacing all GPUs in the cluster is unnecessary and costly unless the issue is confirmed to be widespread. It is not a practical first step for hardware diagnosis.