NCA-GENL Question 190
Select 2You are tasked with writing a script to preprocess a dataset for training a Generative AI Large Language Model (LLM). Under the supervision of a senior team member, you are instructed to implement tokenization and remove unnecessary metadata from the dataset. Which of the following steps should you include in your script to meet the requirements?
- A
Implement a tokenizer to split text into individual tokens.
- B
Remove stopwords and punctuation from the text.
- C
Delete any entries in the dataset containing null or missing values.
- D
Identify and exclude metadata fields that are unrelated to training, such as document IDs or timestamps.
- E
Train the tokenizer from scratch without any pre-trained models.
Show answer and explanation
Correct answers: A, D
Explanation
To meet the requirements of preprocessing data for an LLM, you should focus on tokenization and removing irrelevant metadata. Tokenization splits the text into usable inputs for the model, while excluding unrelated metadata ensures that extraneous information does not interfere with training. Other steps like stopword removal or training a tokenizer from scratch are not explicitly required for this task.
- A. Correct.
Correct. Tokenization is a fundamental preprocessing step for training LLMs, as it converts text into tokens that can be used by the model.
- B. Incorrect.
Incorrect. While removing stopwords and punctuation can be useful for some NLP tasks, it is not always necessary or recommended for training LLMs, as this might remove important context or structure.
- C. Incorrect.
Incorrect. Removing null or missing values is a general data-cleaning step but is not directly related to tokenization or removing metadata as required in this scenario.
- D. Correct.
Correct. Excluding metadata fields that do not contribute to model training, such as document IDs or timestamps, ensures that irrelevant information does not affect the model's learning process.
- E. Incorrect.
Incorrect. Training a tokenizer from scratch is not necessary in this scenario, as you can use pre-trained tokenizers to save time and ensure compatibility with existing LLM architectures.