NCA-GENM Question 82
Select 4You are tasked with evaluating the performance of two generative AI models designed for multimodal tasks, such as image captioning. Model A uses a transformer-based architecture optimized for text-to-image tasks, while Model B uses a convolutional neural network (CNN) combined with a recurrent neural network (RNN) for caption generation. Which evaluation steps and metrics are most appropriate to assess the quality and effectiveness of these models?
- A
Use BLEU or ROUGE scores to measure the textual accuracy of the generated captions.
- B
Evaluate the visual realism of image outputs using Structural Similarity Index (SSIM) or Fréchet Inception Distance (FID).
- C
Perform user studies to gather subjective feedback on caption relevance and image quality.
- D
Measure the training time of each model to determine which architecture is faster.
- E
Check the parameter count of each model to decide which one is more efficient.
- F
Use cross-modal retrieval tasks to test how well the model aligns image and text representations.
Show answer and explanation
Correct answers: A, B, C, F
Explanation
To effectively evaluate generative AI models for multimodal tasks, it is crucial to use a combination of quantitative metrics (e.g., BLEU, FID) and qualitative methods (e.g., user studies). Additionally, cross-modal tasks assess the alignment between modalities, which is central to multimodal applications. Metrics like training time or parameter count, while useful for other purposes, do not directly measure the quality or effectiveness of the models' outputs.
- A. Correct.
BLEU and ROUGE are standard metrics for evaluating the textual quality of generated outputs, such as captions, making them highly relevant for this scenario.
- B. Correct.
SSIM and FID are common metrics for evaluating image quality and realism, which are critical for assessing the visual outputs of a multimodal generative model.
- C. Correct.
User studies provide subjective but valuable insights into the perceived quality and relevance of the generated outputs, complementing quantitative metrics.
- D. Incorrect.
While training time is important for operational efficiency, it is not an appropriate metric for directly evaluating model quality or effectiveness.
- E. Incorrect.
Parameter count relates to model size and efficiency but does not provide insights into the performance or quality of the outputs.
- F. Correct.
Cross-modal retrieval tasks are a strong indicator of how well the model aligns multimodal data, which is a key aspect of generative AI performance in multimodal applications.