Databricks Generative AI Engineer Associate Question 42
Select 3You are working on a Databricks project to fine-tune a generative AI model for text generation. The dataset provided contains missing values, duplicate rows, and irrelevant columns. What steps should you take to prepare the data effectively for training the model?
- A
Remove duplicate rows from the dataset.
- B
Fill missing values with a placeholder token or remove rows with missing values, depending on the use case.
- C
Retain all columns in the dataset, even if they are irrelevant to the model's intended task.
- D
Perform tokenization and normalization on the text data after addressing missing values and duplicates.
- E
Ignore missing values and duplicate rows, as the model can handle them during training.
Show answer and explanation
Correct answers: A, B, D
Explanation
Data preparation is crucial for training generative AI models. Removing duplicates ensures the dataset is not biased, while handling missing values avoids errors during training. Tokenization and normalization are necessary transformations for text data. Retaining irrelevant columns or ignoring data quality issues can introduce noise and hinder model performance, so those options should be avoided.
- A. Correct.
Removing duplicate rows ensures that the model is not biased by redundant data, which could lead to overfitting.
- B. Correct.
Handling missing values appropriately (either filling them with a placeholder or removing them) ensures the dataset is clean and suitable for training.
- C. Incorrect.
Retaining irrelevant columns adds noise to the dataset, which can negatively impact the model's performance and should be avoided.
- D. Correct.
Tokenization and normalization are essential preprocessing steps to convert raw text into a structured format for model training.
- E. Incorrect.
Ignoring missing values and duplicate rows can lead to data quality issues and degrade the model's performance.