NCA-GENL Question 122
Single answerA data scientist is comparing two generative AI language models, Model A and Model B, designed to summarize text. The scientist uses statistical performance metrics including cross-entropy loss and R² (proportion of explained variance) to evaluate the models. Model A has a lower cross-entropy loss than Model B, but Model B has a higher R² score. Based on these metrics, which statement is most accurate for comparing the two models?
- A
Model A is better because lower cross-entropy loss indicates it predicts more accurate probabilities.
- B
Model B is better because a higher R² score means it explains more variance in the data.
- C
Both metrics must be considered together, as cross-entropy loss measures predictive accuracy while R² evaluates variance explained.
- D
Neither metric is useful for comparing generative AI models, as they are designed for regression tasks only.
Show answer and explanation
Correct answer: C
Explanation
When comparing models, it is essential to consider multiple performance metrics as each provides unique insights. Cross-entropy loss measures how well the model predicts probabilities, which is critical for tasks like text generation. R², on the other hand, evaluates how much variance in the target data the model explains. By analyzing both metrics, a more holistic understanding of model performance can be achieved.
- A. Incorrect.
Cross-entropy loss is indeed a critical metric for evaluating the predictive accuracy of probability distributions, but it does not account for how well the model explains variance in the data.
- B. Incorrect.
While R² is useful for understanding how much variance is explained by the model, it does not fully capture performance for generative tasks like summarization, where predictive accuracy is crucial.
- C. Correct.
This is correct because both cross-entropy loss and R² provide complementary insights: one measures predictive accuracy (cross-entropy loss), while the other evaluates how well the model accounts for variance (R²). Together, they offer a broader evaluation of model performance.
- D. Incorrect.
This is incorrect because both cross-entropy loss and R² are applicable to evaluating generative language models, even though they originated from other contexts. They provide valuable insights into different aspects of model performance.