NCA-GENL Question 174
Select 3A company is developing a large-scale generative AI application that leverages an LLM to provide real-time text generation for customer support. The application must handle high volumes of concurrent requests with low latency. Which combination of system components would best meet the user needs?
- A
High-performance GPUs optimized for AI workloads, such as NVIDIA A100 or H100
- B
A single CPU-based server with high clock speed and large memory
- C
A scalable storage solution with high IOPS and low latency, such as NVMe SSDs
- D
A software stack including frameworks like NVIDIA Triton Inference Server for optimized model deployment
- E
An edge device with limited compute for cost-saving purposes
Show answer and explanation
Correct answers: A, C, D
Explanation
To meet the user needs of real-time text generation with high concurrency and low latency, the system must include high-performance GPUs for computational power, scalable and fast storage for data access, and a robust software stack like NVIDIA Triton Inference Server for optimizing inference. CPUs and edge devices with limited compute are insufficient for such demanding workloads.
- A. Correct.
High-performance GPUs like NVIDIA A100 or H100 are specifically designed for AI workloads, including LLM inference. They provide the necessary computational power to handle large-scale generative AI models efficiently.
- B. Incorrect.
While a high-clock-speed CPU with large memory may be useful for certain tasks, it is insufficient for the parallel processing demands of LLM inference, especially in high-concurrency scenarios.
- C. Correct.
A scalable storage solution with high IOPS and low latency ensures that the system can handle large datasets and provides fast access to model weights, which is critical for reducing inference latency.
- D. Correct.
NVIDIA Triton Inference Server is designed to optimize model deployment and inference performance, making it an essential software component for real-time LLM applications.
- E. Incorrect.
An edge device with limited compute may reduce costs but lacks the computational resources to handle the high demands of large-scale LLM inference, especially in a real-time, high-concurrency environment.