NCA-GENL Question 123
Single answerYou are evaluating the performance of two large language models (LLMs) trained for text summarization. Model A achieves a lower cross-entropy loss but a lower R-squared (proportion of explained variance) compared to Model B. Which model would you recommend if the priority is to ensure the model generates summaries as close as possible to the ground truth while maintaining interpretability of variance?
- A
Model A, because a lower cross-entropy loss indicates it generates more accurate summaries.
- B
Model B, because a higher R-squared suggests it explains more variance in the data.
- C
Model A, because cross-entropy loss is more important than R-squared for summarization tasks.
- D
Model B, because R-squared is a more reliable metric than cross-entropy loss for evaluating LLMs.
Show answer and explanation
Correct answer: A
Explanation
Cross-entropy loss is a key metric for evaluating text generation tasks like summarization, as it directly measures how well the predicted outputs match the ground truth. While R-squared is a valuable metric for understanding the proportion of variance explained by the model, it is less relevant for assessing the accuracy of text outputs. Therefore, Model A's lower cross-entropy loss makes it the better choice for this task.
- A. Correct.
Correct. Cross-entropy loss measures how close the generated summaries are to the ground truth. A lower value indicates better performance for this task, where accuracy of generated text is prioritized.
- B. Incorrect.
Incorrect. While a higher R-squared indicates the model explains more variance, cross-entropy loss is more directly tied to the accuracy of the generated summaries in this context.
- C. Incorrect.
Incorrect. Cross-entropy loss is indeed important, but the justification here ignores the role of R-squared and the task-specific priority.
- D. Incorrect.
Incorrect. R-squared is useful for understanding variance but is not the primary metric for evaluating text summarization accuracy in LLMs.