NCA-GENL Question 113
Select 3You are tasked with evaluating a generative AI model that was fine-tuned using Reinforcement Learning from Human Feedback (RLHF). The goal is to assess the model's performance on a customer support chatbot. Which of the following steps are essential for conducting a proper evaluation of the model's effectiveness and ensuring ethical use of human subjects?
- A
Define clear and measurable evaluation metrics, such as response accuracy and user satisfaction.
- B
Ensure that all human participants involved in labeling tasks have given informed consent.
- C
Avoid measuring user satisfaction to prevent subjective bias in the evaluation process.
- D
Conduct a pre-evaluation to test the reliability of the labeling guidelines provided to human annotators.
- E
Allow the model to self-evaluate its output to reduce reliance on human feedback.
Show answer and explanation
Correct answers: A, B, D
Explanation
To properly evaluate a generative AI model fine-tuned with RLHF, it is essential to define clear evaluation metrics, ensure ethical treatment of human participants, and test labeling guidelines beforehand to ensure data reliability. Metrics like user satisfaction and response accuracy are critical, and human judgment is indispensable for alignment. Neglecting these steps can lead to biased or unethical outcomes.
- A. Correct.
Defining clear and measurable evaluation metrics is critical to objectively assess the performance of the AI model, especially in tasks such as response accuracy and user satisfaction.
- B. Correct.
Informed consent is an ethical requirement when involving human participants in labeling or reinforcement learning tasks to ensure they understand the purpose and scope of their contributions.
- C. Incorrect.
Avoiding user satisfaction measurements would hinder a comprehensive evaluation, as user satisfaction is a key indicator of the model's practical effectiveness in customer support scenarios.
- D. Correct.
Conducting a pre-evaluation ensures that labeling guidelines are clear and reliable, reducing ambiguity for human annotators and improving the quality of training data.
- E. Incorrect.
Allowing the model to self-evaluate its output lacks the human perspective required to ensure alignment with user expectations and ethical considerations.