Databricks Generative AI Engineer Associate Question 38
Select 3You are preparing a dataset for training a generative AI model in Databricks. The dataset contains text data with inconsistent formatting, missing values, and unnecessary special characters. Which of the following steps should you perform to ensure the data is clean and ready for training?
- A
Remove special characters and unnecessary whitespace from the text data.
- B
Fill missing values with placeholder tokens or impute them using statistical methods.
- C
Normalize numerical features in the dataset to a standard scale.
- D
Convert all text data to lowercase to ensure consistency.
- E
Directly feed the raw dataset into the model without transformations to preserve original data.
Show answer and explanation
Correct answers: A, B, D
Explanation
Preparing a dataset for training a generative AI model involves cleaning and preprocessing the data to ensure it is consistent and usable. For text data, this typically includes removing special characters and whitespace, handling missing values, and ensuring uniform formatting (e.g., converting to lowercase). These steps help improve model performance by reducing noise and inconsistencies in the input data.
- A. Correct.
Removing special characters and unnecessary whitespace is a standard preprocessing step for text data to clean it and make it consistent for model input.
- B. Correct.
Handling missing values is essential for ensuring the dataset is complete and usable by the model. Filling missing values with placeholder tokens or imputation is a common technique in NLP tasks.
- C. Incorrect.
While normalizing numerical features is important for numerical datasets, it is not directly relevant to text data preparation for generative AI models.
- D. Correct.
Converting text to lowercase ensures uniformity and reduces the vocabulary size, which is critical for efficient language modeling.
- E. Incorrect.
Feeding raw data directly into the model without transformations is not advisable, as it may contain inconsistencies or noise that could degrade the model's performance.