NCA-GENL Question 71
Select 3You are tasked with preparing a dataset for training a generative AI language model. During the data preparation process, you notice that some text samples contain highly imbalanced distributions of rare tokens, while others include significant noise such as HTML tags and non-language characters. Which steps should you prioritize to ensure the dataset is clean and suitable for model training?
- A
Remove non-language characters and HTML tags from the dataset.
- B
Balance the distribution of rare tokens by oversampling sentences containing these tokens.
- C
Normalize text data by converting it to lowercase and removing extra whitespace.
- D
Remove all sentences containing rare tokens to simplify the dataset.
- E
Tokenize the dataset using a pre-trained tokenizer tailored to the language model.
Show answer and explanation
Correct answers: A, C, E
Explanation
Proper inspection, cleansing, and transformation of data are essential for preparing a high-quality dataset for generative AI models. Removing noise (e.g., HTML tags and non-language characters) ensures the dataset's integrity. Normalizing text data enhances uniformity, while tokenization prepares the data for model input. Balancing rare tokens artificially or outright removing them could harm the dataset's natural distribution or diversity, leading to a suboptimal model.
- A. Correct.
Removing non-language characters and HTML tags is crucial for cleaning noisy data and ensuring that only meaningful text is fed into the model.
- B. Incorrect.
Oversampling sentences with rare tokens can introduce bias, distorting the natural distribution of the language and leading to suboptimal model performance.
- C. Correct.
Normalizing text data (e.g., converting to lowercase and removing extra whitespace) ensures consistency and reduces redundancy during training.
- D. Incorrect.
Removing all sentences with rare tokens can lead to a loss of valuable information and diminish the dataset's diversity, which is critical for training a generative model.
- E. Correct.
Tokenizing the dataset with a pre-trained tokenizer ensures that text is converted into a format the model can understand, aligning with the architecture's requirements.