AIF-C01 Question 149
Select 2A company is using a foundation model for a natural language processing (NLP) task, such as text summarization. They want to evaluate the model's performance to ensure it meets their quality standards. Which methods are appropriate for evaluating the performance of the foundation model?
- A
Use BLEU or ROUGE scores to compare the model's output with reference summaries.
- B
Assess the model’s performance by monitoring its GPU utilization during inference.
- C
Conduct human evaluation by having domain experts rate the quality of the summaries.
- D
Measure the model's accuracy by comparing it to a benchmark dataset for classification tasks.
- E
Evaluate the model's performance using prompt engineering to improve outputs.
Show answer and explanation
Correct answers: A, C
Explanation
Evaluating foundation model performance for tasks like summarization often involves objective metrics such as BLEU and ROUGE, as well as subjective methods like human evaluation. These approaches measure the quality and relevance of the generated summaries. Other methods, such as monitoring GPU utilization or using metrics like accuracy for unrelated tasks, are not suitable for this purpose.
- A. Correct.
BLEU and ROUGE are standard metrics used for evaluating the quality of text generation tasks like summarization. They compare the model's output with reference summaries, making this an appropriate method.
- B. Incorrect.
Monitoring GPU utilization is a measure of resource efficiency, not a performance metric for evaluating the quality of the model's outputs.
- C. Correct.
Human evaluation is a reliable method for assessing the quality of text generation tasks, especially when subjective judgment is required, such as in summarization.
- D. Incorrect.
Accuracy is commonly used for classification tasks, but it is not suitable for evaluating a summarization task, which involves generating text rather than predicting labels.
- E. Incorrect.
Prompt engineering is a method for improving the model's outputs, but it is not an evaluation method to measure the model's performance.