SnowPro Specialty: Gen AI Question 276
Select 2Evaluation metricsA retail company uses a Snowflake-hosted retrieval-augmented generation (RAG) application to answer customer support questions from its product manuals and return policy documents. In testing, stakeholders notice two recurring issues: some answers sound fluent but include policy details not found in the retrieved documents, and other answers omit important return conditions even when the correct documents were retrieved. The team wants to choose evaluation metrics that best distinguish these failure modes before promoting the application to production. Which TWO metrics are the most appropriate?
- A
Groundedness, to measure whether the generated answer is supported by the retrieved context
- B
Answer relevance/completeness, to measure whether the response fully addresses the user question using the available context
- C
Warehouse credit consumption, to measure whether the model produces lower-cost answers
- D
Latency, to measure whether the answer is generated quickly enough for interactive use
- E
Token count, to measure whether the answer contains enough detail to be correct
Show answer and explanation
Correct answers: A, B
Explanation
This scenario separates two common RAG evaluation problems: unsupported generation and incomplete generation. Groundedness evaluates whether the answer is supported by the retrieved context, making it useful for detecting hallucinations relative to source documents. Answer relevance or completeness evaluates whether the model actually answers the question and includes the necessary details when the evidence is available. Together, these metrics help teams diagnose whether failures come from the generation step rather than from retrieval alone. By contrast, latency and cost are operational metrics, and token count is only a superficial text property. Snowflake best practices for GenAI evaluation emphasize using task-appropriate quality metrics for LLM applications, especially in RAG workflows, where faithfulness to context and usefulness of the final answer are distinct dimensions that should be evaluated separately.
- A. Correct.
Correct. Groundedness is the right metric for detecting when an answer is fluent but introduces unsupported claims or policy details not present in the retrieved documents. In RAG systems, this helps identify hallucinations relative to the supplied context, which is one of the exact issues described in the scenario.
- B. Correct.
Correct. Answer relevance or completeness is appropriate for cases where the correct documents were retrieved but the generated answer still misses important conditions. This metric evaluates whether the response actually addresses the user's question sufficiently and covers the necessary information from the available context.
- C. Incorrect.
Incorrect. Warehouse credit consumption is an operational cost metric, not a quality evaluation metric for LLM output. It may matter for deployment decisions, but it does not distinguish between unsupported claims and incomplete answers.
- D. Incorrect.
Incorrect. Latency is an important system performance metric, especially for user experience, but it does not measure factual support or completeness of generated content. A fast answer can still be hallucinated or incomplete.
- E. Incorrect.
Incorrect. Token count is not a reliable quality metric. Longer answers are not necessarily more correct, and shorter answers are not necessarily incomplete. Using token count as a proxy for correctness or completeness is a common but flawed assumption.