NCA-GENL Question 91
Single answerYou are working as an associate in a team building a generative AI model. Under the supervision of a senior team member, you are tasked with analyzing the training dataset. During the analysis, you notice that the dataset contains a significant number of duplicate entries. What is the most appropriate action to take in this scenario?
- A
Remove all duplicate entries immediately to ensure data cleanliness.
- B
Consult the senior team member to understand whether the duplicates are intentional and their potential impact on the model.
- C
Proceed with training the model without addressing the duplicates, as modern LLMs can handle such issues automatically.
- D
Replace duplicate entries with synthetic data to increase dataset variability.
Show answer and explanation
Correct answer: B
Explanation
When conducting data analysis for generative AI under the supervision of a senior team member, it is critical to consult them regarding significant observations like duplicate entries. Duplicates might be intentional for specific purposes, or they could negatively impact model performance. Seeking guidance ensures the dataset is handled correctly and aligns with project goals.
- A. Incorrect.
Removing all duplicate entries immediately might harm the dataset if the duplicates were intentionally included for specific reasons, such as balancing the dataset. This action should only be taken after consultation.
- B. Correct.
Consulting the senior team member is the correct approach, as they can provide guidance on whether the duplicates are intentional and how they might affect the training process. This ensures that the dataset is treated appropriately.
- C. Incorrect.
Proceeding with training without addressing the duplicates is not advisable, as duplicates can lead to issues like overfitting or bias in the model. Modern LLMs are not inherently designed to handle all issues caused by duplicate data.
- D. Incorrect.
Replacing duplicate entries with synthetic data is not a recommended approach unless explicitly instructed by the senior team member. Synthetic data generation requires careful planning and validation to ensure it does not introduce new biases.