NCA-GENL Question 146
Select 3You are tasked with fine-tuning a large language model (LLM) for a customer service chatbot application. After creating the fine-tuned model, you want to ensure the software is reliable and performs effectively in production. Which steps should you take to maintain and test the model before deployment?
- A
Evaluate the model using domain-specific datasets to test its performance on customer inquiries.
- B
Implement continuous monitoring of the model’s outputs in production to detect drift or errors.
- C
Disable logging to avoid collecting sensitive customer data during testing.
- D
Conduct load testing to assess the model’s performance under high-traffic scenarios.
- E
Retrain the model daily, regardless of performance, to ensure it remains up to date.
Show answer and explanation
Correct answers: A, B, D
Explanation
Maintaining and testing an LLM before deployment involves validating its performance using domain-specific data, ensuring it can handle production-level traffic, and implementing monitoring for long-term reliability. Disabling logging is counterproductive, and retraining should be done based on performance needs rather than arbitrarily.
- A. Correct.
Evaluating the model using domain-specific datasets is essential to ensure it performs well on customer-specific inquiries and aligns with the application requirements.
- B. Correct.
Continuous monitoring in production helps detect model drift or errors, ensuring the model remains reliable over time.
- C. Incorrect.
Disabling logging altogether is not a good practice as it prevents effective debugging and analysis. Instead, techniques like anonymization or encryption should be used to safeguard sensitive data.
- D. Correct.
Load testing is critical to understanding how the model performs under real-world high-traffic scenarios and ensures it can handle production workloads.
- E. Incorrect.
Retraining the model daily without assessing its performance or necessity is an inefficient use of resources and may not provide meaningful improvements.