NCA-GENL Question 70
Select 3You are working on training a generative AI language model for a customer support chatbot. During the data preprocessing phase, you notice that the dataset contains numerous missing values, inconsistent formats (e.g., dates in multiple formats), and duplicate entries. Which of the following actions should you take to ensure the data is properly prepared for training the model?
- A
Remove duplicate entries to prevent biased training results.
- B
Use tokenization to directly handle missing values during training.
- C
Standardize inconsistent formats, such as normalizing date formats.
- D
Impute missing values using appropriate strategies like mean, median, or mode.
- E
Leave duplicate entries intact as they may improve the model's performance.
Show answer and explanation
Correct answers: A, C, D
Explanation
Proper data preprocessing is critical for training generative AI models. This includes removing duplicates to avoid bias, standardizing formats for consistency, and imputing missing values to ensure data completeness. These steps ensure the model is trained on high-quality, clean data, which is essential for accurate and reliable performance.
- A. Correct.
Removing duplicate entries is crucial to avoid introducing bias and redundancy in the training dataset. Duplicate data can distort the model's learning process.
- B. Incorrect.
Tokenization is a process for breaking down text into smaller units and does not address missing values. Handling missing values requires separate preprocessing steps.
- C. Correct.
Standardizing inconsistent formats, such as transforming all dates into a unified format, ensures the data is clean and interpretable by the model.
- D. Correct.
Imputing missing values with appropriate strategies ensures the dataset remains complete and usable for training the model. This is a common data preprocessing step.
- E. Incorrect.
Leaving duplicate entries intact is not recommended, as it can lead to biased training results and overfitting to redundant patterns in the data.