Databricks Generative AI Engineer Associate Question 39
Select 3You are tasked with preparing a dataset for training a generative AI model on Databricks. The dataset contains millions of unstructured text files in various formats, such as JSON, CSV, and plain text. To ensure the data is usable for training, you need to standardize it into a single format and clean it by removing unwanted characters. Which combination of steps should you take to efficiently prepare the dataset on Databricks?
- A
Use Databricks Auto Loader to ingest the files into a Delta table for efficient processing.
- B
Use Spark SQL to directly train the generative AI model on the raw unstructured files.
- C
Apply Spark transformations to clean the data, such as removing special characters and duplicates.
- D
Convert all text data into a uniform format, such as Parquet or Delta, for optimized storage and querying.
- E
Directly feed the raw text files to the model without cleaning or standardization since generative AI models handle noise well.
Show answer and explanation
Correct answers: A, C, D
Explanation
Preparing data for generative AI involves several key steps: ingesting data efficiently using tools like Databricks Auto Loader, cleaning and standardizing the data to remove unwanted noise, and converting it into a format optimized for storage and querying. These steps ensure that the data is ready for effective model training. Simply feeding raw, unprocessed data to a model is not a best practice.
- A. Correct.
Correct. Databricks Auto Loader is a scalable and efficient tool for ingesting large datasets into a Delta table, which enables efficient data processing.
- B. Incorrect.
Incorrect. Spark SQL is not designed for training models directly on raw unstructured data. Preprocessing and standardization are crucial before model training.
- C. Correct.
Correct. Cleaning the data using Spark transformations ensures that the dataset is ready for training and does not contain noise or inconsistencies.
- D. Correct.
Correct. Converting the data into a uniform format like Parquet or Delta ensures optimized storage and provides faster querying capabilities during the training process.
- E. Incorrect.
Incorrect. Raw text files often contain noise and inconsistencies. Feeding them directly to a model without preprocessing can lead to suboptimal performance.