NCA-GENL Question 90
Single answerYou are evaluating two generative AI models trained on a text dataset. Model A has a lower cross-entropy loss than Model B, but Model B has a higher R-squared (proportion of explained variance) when predicting token probabilities. Which of the following statements best explains which model is better?
- A
Model A is better because a lower cross-entropy loss indicates better performance.
- B
Model B is better because a higher R-squared indicates better performance.
- C
It depends on the use case; cross-entropy loss measures predictive accuracy, while R-squared measures how well the model explains variation in the data.
- D
Neither model is better because these metrics are not suitable for evaluating generative models.
Show answer and explanation
Correct answer: C
Explanation
When comparing models using statistical performance metrics, it is important to consider the specific use case and the goals of the evaluation. Cross-entropy loss measures how accurate the model’s predictions are, which is critical for tasks like language modeling. R-squared, on the other hand, evaluates how much of the variance in the data is explained by the model, which might be more relevant for regression-like tasks. Neither metric is inherently better; their utility depends on the context in which the model is being evaluated.
- A. Incorrect.
Cross-entropy loss is indeed a key metric for evaluating generative AI models, but it alone does not determine which model is better without considering the use case or other metrics.
- B. Incorrect.
R-squared is useful for understanding how well a model explains variance in the data, but it does not always align with the goals of generative AI, especially for probabilistic modeling.
- C. Correct.
This is the correct answer because the choice of the best model depends on the specific use case. Cross-entropy loss evaluates predictive accuracy, while R-squared measures the proportion of variance explained by the model. These metrics serve different purposes.
- D. Incorrect.
This is incorrect because cross-entropy loss and R-squared are valid metrics for evaluating generative models, although their relevance depends on the context.