NCA-GENL Question 88
Single answerYou are tasked with comparing two large language models (LLMs) trained on the same dataset to determine which one performs better. Model A has a lower cross-entropy loss, while Model B has a higher R² (proportion of explained variance). Which metric would you prioritize to evaluate their performance, and why?
- A
Prioritize cross-entropy loss because it directly measures how well the model predicts the next token in sequence tasks.
- B
Prioritize R² because it measures how well the model explains variance across a regression task, which is more important for language models.
- C
Use both metrics equally, as they measure different aspects of model performance and provide complementary insights.
- D
Neither metric is useful, as they do not apply to evaluating large language models.
Show answer and explanation
Correct answer: A
Explanation
Cross-entropy loss is the most appropriate metric for evaluating large language models because it directly measures the model's ability to predict the next token, which is the primary objective in sequence modeling tasks. R², on the other hand, is more relevant for regression problems and does not effectively capture the predictive capabilities of LLMs in text generation tasks.
- A. Correct.
Cross-entropy loss is a commonly used loss function for sequence modeling tasks like language modeling. A lower cross-entropy loss indicates that the model is better at predicting the next token, which is a critical task for LLMs.
- B. Incorrect.
While R² is a useful metric for regression tasks, it is not typically used to evaluate large language models, as their primary goal is not to explain variance in regression tasks but to predict sequences of text.
- C. Incorrect.
Although using multiple metrics can sometimes provide complementary insights, in this case, cross-entropy loss is a more specific and appropriate metric for evaluating LLM performance compared to R².
- D. Incorrect.
This is incorrect because cross-entropy loss is directly applicable to LLM evaluation, and R² can be used in specific contexts, even if not ideal for LLMs. Both metrics have relevance depending on the task.