NCA-GENL Question 105
Select 3A team is training a large language model (LLM) for a healthcare application. During the evaluation phase, they notice that the model’s accuracy on a specific test dataset is significantly lower than expected. Which factors should the team investigate to identify relationships or trends that could be affecting their results?
- A
The distribution of data in the training and test sets
- B
The optimization algorithm used during model training
- C
The presence of domain-specific terminology in the test dataset
- D
The availability of GPU resources during inference
- E
The hyperparameter values used during training
Show answer and explanation
Correct answers: A, C, E
Explanation
To identify relationships and trends affecting the results, it is important to examine factors like data distribution, the presence of domain-specific terms, and hyperparameter settings, as these directly impact the model's ability to generalize and perform well on test datasets. Other factors, like inference hardware, are less likely to explain accuracy issues.
- A. Correct.
If the data distribution between training and test sets does not align, the model may struggle to generalize, leading to poor performance on the test data.
- B. Incorrect.
While the optimization algorithm is important for training, it is less likely to directly impact results during evaluation unless it causes instability during training.
- C. Correct.
Domain-specific terminology in the test dataset could affect the model's accuracy if such terms were underrepresented or absent in the training data.
- D. Incorrect.
GPU resources during inference primarily affect speed and performance but are unlikely to directly impact accuracy or results.
- E. Correct.
Hyperparameter values influence the training process and could lead to underfitting or overfitting, affecting the model's ability to generalize to test data.