Databricks Generative AI Engineer Associate Question 40
Select 4You are preparing a dataset for training a generative AI model in Databricks. The dataset contains text data with inconsistent formatting, such as mixed casing, extra whitespace, and special characters. Which of the following steps are most appropriate for preparing the data for training?
- A
Normalize text by converting it to lowercase.
- B
Remove extra whitespace and special characters.
- C
Manually label all data points to ensure high-quality training samples.
- D
Split the dataset into training, validation, and test sets.
- E
Apply tokenization to convert text into numerical format.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Data preparation for training a generative AI model involves standardizing and cleaning the input data (e.g., normalizing text, removing noise), splitting the data into appropriate subsets for training, validation, and testing, and converting text into a numerical format using techniques like tokenization. These steps ensure the model can learn effectively and generalize well to unseen data.
- A. Correct.
Converting text to lowercase ensures consistency, making it easier for the model to learn without being affected by case differences.
- B. Correct.
Removing extra whitespace and special characters eliminates noise and improves the quality of the data.
- C. Incorrect.
Manually labeling all data is not relevant to this scenario, as the question focuses on preparing the data rather than labeling it.
- D. Correct.
Splitting the dataset is essential for evaluating the model's performance and preventing overfitting.
- E. Correct.
Tokenization is necessary to convert raw text data into a format that the model can process.