NCA-GENL Question 114
Select 3A team is evaluating a generative AI model fine-tuned on labeled data with reinforcement learning from human feedback (RLHF). During the evaluation, they observe inconsistencies between model outputs and human expectations. Which of the following actions would most effectively help improve the model's alignment with human preferences?
- A
Increase the size of the training dataset used in the initial pretraining phase.
- B
Refine the reward model used in RLHF to better represent human preferences.
- C
Conduct a thorough evaluation of the quality and consistency of human-labeled data used in RLHF.
- D
Increase the number of layers in the model architecture to enhance its complexity.
- E
Incorporate more diverse human feedback by involving a larger pool of annotators during RLHF.
Show answer and explanation
Correct answers: B, C, E
Explanation
To improve alignment in a generative AI model trained using RLHF, it is crucial to focus on refining the reward model, ensuring the quality of human-labeled data, and incorporating diverse feedback. These steps directly address the alignment process by improving the inputs and signals used during the RLHF process, whereas changes to pretraining or model complexity do not specifically target alignment issues.
- A. Incorrect.
Increasing the training dataset size for the initial pretraining phase improves generalization but does not directly address alignment issues observed during RLHF.
- B. Correct.
Refining the reward model to better capture human preferences directly impacts how the model aligns with human expectations, making this a critical step in RLHF improvement.
- C. Correct.
Ensuring the human-labeled data is consistent and high-quality is essential for effective RLHF since poorly labeled data can misguide the model's learning process.
- D. Incorrect.
Increasing the number of layers in the model architecture might improve its capacity but does not directly resolve alignment issues caused by RLHF or human feedback inconsistencies.
- E. Correct.
Incorporating more diverse human feedback helps reduce biases and ensures that the model learns to align with a broader range of human preferences.