Databricks Generative AI Engineer Associate Question 233
Single answerYou are tasked with deploying a Large Language Model (LLM) application that leverages a Foundation Model API, such as OpenAI's GPT or Hugging Face's hosted models. The application must serve multiple concurrent users, offer low latency, and handle scalability effectively. What is the best approach to serve this application in a production environment?
- A
Deploy the application on a serverless function platform and make synchronous calls to the Foundation Model API for each user request.
- B
Use a scalable compute cluster (e.g., Databricks clusters) and implement caching for common queries to reduce API calls.
- C
Run the Foundation Model locally within your infrastructure to eliminate the need for external API calls.
- D
Deploy the application on a single virtual machine and rely on increasing its hardware capacity as user demand grows.
Show answer and explanation
Correct answer: B
Explanation
The correct approach for serving an LLM application leveraging Foundation Model APIs involves using scalable infrastructure, such as Databricks clusters, and optimizing performance through techniques like caching. This ensures the application can handle high concurrency, maintain low latency, and scale efficiently as user demand increases.
- A. Incorrect.
Serverless functions are not ideal for serving LLM applications with high concurrency and low latency requirements because they can introduce cold start delays and lack fine-grained control over performance tuning.
- B. Correct.
Using a scalable compute cluster like Databricks allows for horizontal scaling and efficient management of API calls. Caching common queries further improves performance and reduces latency.
- C. Incorrect.
Running a Foundation Model locally can be extremely resource-intensive and may not be feasible due to hardware constraints and licensing restrictions for many hosted models.
- D. Incorrect.
A single virtual machine is not scalable and becomes a bottleneck for handling increasing user demand, making it unsuitable for production environments with high traffic.