NCA-GENL Question 85
Single answerYou are tasked with comparing two generative AI language models (Model A and Model B) for a text-generation task. Model A has a lower cross-entropy loss than Model B on the validation dataset, while Model B demonstrates a higher R² (proportion of explained variance) when evaluated on a set of regression-based metrics. Which model would you prioritize for the task, and why?
- A
Model A, because a lower cross-entropy loss indicates better predictive performance for text generation.
- B
Model A, because R² is not relevant for text-generation tasks.
- C
Model B, because a higher R² indicates better performance across all types of generative tasks.
- D
Model B, because a higher R² may indicate better contextual understanding, depending on the specific task requirements.
Show answer and explanation
Correct answer: A
Explanation
For a text-generation task, cross-entropy loss is a more task-appropriate metric than R² (proportion of explained variance). Cross-entropy directly evaluates how well a language model predicts the next token in a sequence, making it the more relevant measure of performance for generative tasks. While R² can provide insights into regression or other numerical prediction tasks, it is not as informative for text generation.
- A. Correct.
Correct. Cross-entropy loss is the most relevant metric for evaluating generative language models in text-generation tasks, as it directly measures how well the predicted probability distribution matches the true distribution.
- B. Incorrect.
Incorrect. While R² may be less relevant in this context, the decision should primarily be based on the task-specific metrics like cross-entropy loss rather than dismissing R² entirely.
- C. Incorrect.
Incorrect. A higher R² does not necessarily translate to better performance for generative tasks, especially when the task involves text generation. R² is more relevant for regression-based tasks.
- D. Incorrect.
Incorrect. While R² might provide insights into certain model behaviors, it is not the primary metric for evaluating text-generation tasks. Cross-entropy loss is more relevant for this use case.