SnowPro Specialty: Gen AI Question 163
Single answerLatency (for example, fine-tuning, model size)A retail company built a customer-support assistant in Snowflake that uses a large language model to generate replies during live chat sessions. Agents report that response times are too slow during peak hours, causing a poor user experience. The team has already confirmed that the retrieval step is fast and that the main delay comes from model inference. They want to reduce latency without redesigning the entire application. Which action is the MOST appropriate to try first?
- A
Switch to a smaller model that still meets quality requirements for the support use case
- B
Increase the number of documents retrieved for each prompt so the model has more context to reason over
- C
Fine-tune the largest available model because a larger customized model will usually return answers faster
- D
Add more structured fields to the source tables so Snowflake can automatically reduce inference latency
Show answer and explanation
Correct answer: A
Explanation
When inference is the primary latency bottleneck in a generative AI application, one of the best first optimizations is to evaluate a smaller model that still meets the task's quality requirements. This aligns with general LLM best practices: latency and cost tend to increase with model size, so right-sizing the model is often more effective than adding complexity elsewhere. Fine-tuning can improve task performance or consistency, but it should not be treated as a primary latency-reduction technique. Likewise, increasing retrieved context often increases token volume and can worsen latency. In Snowflake-based GenAI solutions, practitioners should identify the actual bottleneck first, retrieval, prompt size, or model inference, and then optimize the component responsible for the delay.
- A. Correct.
Correct. Model size is one of the most direct drivers of inference latency. If the current model is larger than necessary, moving to a smaller model that still satisfies accuracy and quality targets is often the fastest and lowest-risk optimization. In real-world GenAI systems, teams commonly evaluate the smallest model that can meet the business requirement because smaller models usually produce responses faster and at lower cost.
- B. Incorrect.
Incorrect. Retrieving more documents typically increases prompt size, which can increase total processing time and may even worsen end-to-end latency. Since the scenario states that retrieval is already fast and inference is the bottleneck, adding more context is more likely to slow the model down than speed it up. This option reflects the common misconception that more context always improves performance without tradeoffs.
- C. Incorrect.
Incorrect. Fine-tuning may help a model perform better on domain-specific tasks, but it does not inherently make inference faster. In many cases, a larger model remains slower than a smaller one even after customization. This option mixes two separate ideas: model quality/task adaptation and latency optimization. The key issue in the scenario is inference speed, not lack of domain alignment.
- D. Incorrect.
Incorrect. Improving table structure can help data quality, governance, and some upstream pipeline operations, but it does not automatically reduce the inference time of an LLM. Because the delay is explicitly identified as model inference rather than data retrieval or SQL execution, changing source table structure is not the most appropriate first step.