NCA-GENL Question 110
Single answerWhile evaluating a generative AI model trained using Reinforcement Learning from Human Feedback (RLHF), a team notices that the model produces outputs that align well with human preferences but occasionally fails on objective metrics such as factual accuracy. What is the most effective approach to address this issue while maintaining alignment with human preferences?
- A
Incorporate additional objective metrics into the reward model used during RLHF training.
- B
Reduce the reliance on human feedback and replace it entirely with automated evaluation metrics.
- C
Retrain the model from scratch using only objective ground-truth data to ensure factual accuracy.
- D
Introduce a hybrid evaluation framework that balances human feedback with automated checks for factual accuracy.
Show answer and explanation
Correct answer: D
Explanation
RLHF seeks to align AI models with human preferences, but this can lead to challenges when objective metrics like factual accuracy are also important. To address this, a hybrid evaluation framework that balances human feedback with automated evaluations for factual accuracy is the most effective approach as it ensures both alignment and reliability in output.
- A. Incorrect.
Incorporating additional objective metrics into the reward model may help, but it can dilute the importance of human feedback and may not fully address the trade-off between human preferences and factual accuracy.
- B. Incorrect.
Reducing reliance on human feedback undermines the core idea of RLHF, which is to align the model with human preferences, leading to suboptimal outcomes.
- C. Incorrect.
Retraining the model from scratch using only objective ground-truth data sacrifices the alignment with human preferences, which is the primary goal of RLHF.
- D. Correct.
A hybrid evaluation framework ensures the model maintains alignment with human preferences while addressing factual accuracy issues, providing a balanced and effective solution.