NCP-AII Question 3
Select 4You are managing an NVIDIA DGX system running AI workloads, and notice that training performance has degraded significantly compared to earlier runs. Upon investigation, you find that GPU utilization is low and system logs indicate frequent CPU bottlenecks. What steps should you take to troubleshoot and optimize performance?
- A
Monitor CPU and GPU usage in real-time using tools like NVIDIA System Management Interface (nvidia-smi).
- B
Increase the batch size of the training workload to improve GPU utilization.
- C
Disable GPU Boost to stabilize power consumption and clock frequencies.
- D
Check for competing workloads on the system and prioritize AI training tasks.
- E
Update the GPU drivers and CUDA toolkit to the latest version supported by the application.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Low GPU utilization and CPU bottlenecks are common causes of degraded training performance on NVIDIA systems. A systematic approach to troubleshooting involves monitoring system resource usage, optimizing workload parameters (e.g., batch size), resolving resource contention, and ensuring the software stack (drivers and CUDA) is up-to-date. These steps collectively address potential hardware and software inefficiencies, improving overall training performance.
- A. Correct.
Monitoring CPU and GPU usage in real-time using tools like nvidia-smi can help identify whether the bottleneck is caused by underutilized hardware or imbalanced resource allocation.
- B. Correct.
Increasing the batch size often improves GPU utilization as it allows the GPUs to process larger chunks of data simultaneously, reducing idle time.
- C. Incorrect.
Disabling GPU Boost is not recommended in this scenario as it could reduce GPU performance. GPU Boost dynamically adjusts clock speeds for optimal performance.
- D. Correct.
Checking for competing workloads and prioritizing AI training tasks can alleviate CPU contention and improve overall system performance.
- E. Correct.
Updating the GPU drivers and CUDA toolkit ensures compatibility with the latest features and optimizations, which might resolve performance degradation issues.