SnowPro Specialty: Gen AI Question 164
Single answerLatency (for example, fine-tuning, model size)A product team uses a Snowflake Cortex LLM-powered customer support assistant to generate short suggested replies for agents. During peak hours, agents report that response time is too slow for interactive use. The prompts are brief, the expected output is usually under 80 tokens, and retrieval has already been optimized. The team wants to reduce end-to-end model latency without significantly changing the user experience. Which action is the BEST choice?
- A
Switch to a smaller model that still meets quality requirements for the task
- B
Increase the maximum output token setting substantially so the model can finish generation in fewer passes
- C
Fine-tune the largest available model because fine-tuned models are inherently faster at inference than smaller base models
- D
Add more retrieved context documents to the prompt so the model has more information and can answer faster
Show answer and explanation
Correct answer: A
Explanation
For interactive generative AI applications, latency is strongly influenced by model size, input token count, and output token count. When prompts and outputs are already short and retrieval is optimized, the most effective lever is often selecting a smaller model that still meets business quality thresholds. This aligns with general LLM best practices: minimize unnecessary prompt/context size, constrain generation length, and avoid overprovisioning model capability when the task is narrow. Fine-tuning may improve task fit or reduce prompt engineering overhead, but it does not override the basic latency impact of model size. In Snowflake Cortex-based solution design, candidates should evaluate the tradeoff among quality, speed, and cost, and prefer the smallest suitable model for real-time user-facing scenarios.
- A. Correct.
Correct. Smaller models typically have lower inference latency because they require fewer compute resources per request. For short prompts and short completions, choosing the smallest model that still satisfies accuracy and quality requirements is a standard latency optimization. This is especially appropriate when retrieval is already optimized and the application is interactive.
- B. Incorrect.
Incorrect. Increasing the maximum output token limit does not reduce latency. In practice, allowing more output tokens can increase latency because the model may generate longer responses or have a larger generation budget. Since the use case already expects short replies, a higher token limit works against the latency goal.
- C. Incorrect.
Incorrect. Fine-tuning can improve task performance or reduce prompt complexity in some architectures, but it does not make a large model inherently faster than a smaller model at inference. Model size remains a major driver of latency. This option reflects the misconception that fine-tuning is primarily a latency optimization regardless of model size.
- D. Incorrect.
Incorrect. Adding more retrieved context usually increases prompt size, which can increase processing time and token usage. If retrieval has already been optimized, expanding the context window is more likely to hurt latency than help it, especially for a task that only needs brief responses.