NCP-AII Question 12
Select 3You are responsible for maintaining an AI infrastructure cluster running multiple NVIDIA GPUs. One of the nodes has been flagged for reduced performance during training jobs. Which of the following steps would you take to identify a faulty GPU, card, or power supply on the node?
- A
Run the
nvidia-smicommand to check GPU utilization, memory usage, and error logs. - B
Inspect the system's event logs for power supply-related errors or warnings.
- C
Physically remove all GPUs from the node and test them individually on another system.
- D
Verify the temperature and power limits of the GPUs using monitoring tools.
- E
Reboot the node to resolve the issue without any further diagnostics.
Show answer and explanation
Correct answers: A, B, D
Explanation
Identifying faulty GPUs, cards, or power supplies involves using diagnostic tools like nvidia-smi to examine GPU performance, memory, and error logs. System event logs can highlight power supply-related problems, while monitoring tools can reveal temperature and power limit issues. A structured diagnostic approach ensures that hardware faults are accurately identified without unnecessary steps like physically relocating components or relying on temporary fixes like reboots.
- A. Correct.
Running the
nvidia-smicommand is an essential step to diagnose GPU-related issues. This tool provides insights into GPU utilization, memory errors, and other hardware-specific details that can help identify faulty components. - B. Correct.
Inspecting the system's event logs can provide critical information about power supply issues, such as voltage irregularities or power delivery failures, which could affect GPU performance.
- C. Incorrect.
Physically removing and testing GPUs on another system is a time-consuming and unnecessary first step. Diagnostic tools and logs should be used before considering hardware relocation.
- D. Correct.
Monitoring GPU temperature and power limits can reveal overheating or power throttling, which could indicate hardware or cooling system problems.
- E. Incorrect.
Rebooting the node without diagnostics might temporarily resolve symptoms but does not address the root cause of the issue, making it an ineffective troubleshooting step in this scenario.