NCA-GENM Question 107
Select 3A team is testing a multimodal generative AI model designed to generate text captions for images. Which of the following approaches should the team use to ensure the model's accuracy and effectiveness?
- A
Evaluate the model's text outputs using BLEU or ROUGE scores against reference captions.
- B
Test the model on unseen multimodal datasets to measure its generalization ability.
- C
Measure the model's latency and energy consumption during inference.
- D
Solicit feedback from a diverse group of human evaluators to assess caption relevance and quality.
- E
Ensure the model consistently generates outputs identical to the training data.
Show answer and explanation
Correct answers: A, B, D
Explanation
To ensure accuracy and effectiveness, testing AI models requires quantitative evaluations (e.g., BLEU or ROUGE scores), qualitative assessments (e.g., human feedback), and generalization tests on unseen data. These approaches provide comprehensive insights into the model's performance and capabilities. However, focusing solely on deployment metrics (e.g., latency) or training data replication does not align with the goals of evaluating accuracy and effectiveness.
- A. Correct.
Correct. BLEU and ROUGE scores are standard metrics for evaluating the quality of text generated by AI models, making them suitable for assessing caption accuracy.
- B. Correct.
Correct. Testing on unseen datasets helps verify whether the model can generalize well to new data, an essential part of evaluating its effectiveness.
- C. Incorrect.
Incorrect. While latency and energy consumption are important for deployment considerations, they do not directly measure the accuracy or effectiveness of the model's outputs.
- D. Correct.
Correct. Human feedback is crucial for evaluating the semantic relevance and quality of the generated captions, especially when automated metrics fall short.
- E. Incorrect.
Incorrect. Consistently generating outputs identical to training data indicates overfitting, which is a negative trait and not a measure of model accuracy or effectiveness.