NCA-GENL Question 72
Select 3You are tasked with preparing a dataset for training a generative AI large language model (LLM). The dataset contains missing values, inconsistent formatting, and duplicate entries. Which of the following steps should you take to ensure the data is suitable for training?
- A
Remove duplicate entries to prevent skewed model training.
- B
Fill missing values with random data to avoid disrupting the dataset structure.
- C
Standardize text formatting, such as ensuring consistent casing and punctuation.
- D
Analyze the distribution of missing values to decide whether to remove or impute them.
- E
Ignore the inconsistencies in the dataset as the LLM can handle noisy data effectively.
Show answer and explanation
Correct answers: A, C, D
Explanation
Inspecting and cleansing data is crucial for training generative AI LLMs. Removing duplicates prevents bias, standardizing formatting ensures consistency, and analyzing missing values helps maintain data quality. These steps collectively ensure the model receives accurate and well-structured input, improving training outcomes.
- A. Correct.
Removing duplicate entries ensures that the model does not overfit or become biased toward repeated data points, which could lead to skewed training results.
- B. Incorrect.
Filling missing values with random data can introduce noise and bias into the dataset, leading to suboptimal model performance.
- C. Correct.
Standardizing text formatting helps the model recognize patterns more effectively and reduces unnecessary variability in the data.
- D. Correct.
Analyzing the distribution of missing values allows for informed decisions on whether to remove or impute them, maintaining the integrity of the dataset.
- E. Incorrect.
Ignoring inconsistencies can lead to poor model performance, as LLMs can struggle with noisy or improperly prepared data.