NCA-GENL Question 86
Single answerYou are comparing the performance of two generative AI language models, Model A and Model B, designed for text summarization. Model A has a lower mean squared error (MSE) on the test set, while Model B has a higher R² (coefficient of determination). Which model is likely better for this task, and why?
- A
Model A, because a lower MSE indicates less prediction error.
- B
Model B, because a higher R² shows the model explains more variance in the data.
- C
Model A, because MSE is the only metric that matters for evaluating generative AI.
- D
Neither model can be definitively determined as better without considering task-specific priorities.
Show answer and explanation
Correct answer: D
Explanation
When comparing models using statistical performance metrics, such as MSE and R², it is essential to consider the specific requirements and priorities of the task. MSE measures prediction error, which is crucial for accuracy, while R² measures the proportion of variance explained, which is valuable for understanding the model's explanatory power. In many cases, neither metric alone can definitively determine the better model, and a combination of metrics and task-specific goals must guide the evaluation.
- A. Incorrect.
A lower MSE does indicate less prediction error, but it does not necessarily mean Model A is better without considering other metrics like R² and the specific task priorities.
- B. Incorrect.
While a higher R² is desirable for explaining variance, it is not sufficient alone to determine the better model for this task without considering other factors like MSE and the task's specific requirements.
- C. Incorrect.
This is incorrect because MSE is not the only metric relevant for evaluating generative AI models. R² and other metrics like BLEU or ROUGE scores (depending on the task) also play a role.
- D. Correct.
This is correct because the choice of the better model depends on task-specific priorities. For example, if minimizing error is more critical, Model A might be better. However, if explaining variance and capturing relationships in the data is more important, Model B could be preferred.