Databricks Generative AI Engineer Associate Question 92
Select 3You are tasked with extracting data from a large dataset stored in Delta Lake on Databricks to train a generative AI model. The data retrieval process requires filtering records based on specific timestamps, applying transformations, and aggregating the data efficiently. Which of the following tools or techniques would be most appropriate for this task?
- A
Databricks SQL for querying and filtering the dataset
- B
Delta Live Tables for managing and transforming the data pipeline
- C
MLflow for tracking model training parameters
- D
Apache Spark DataFrame API for data transformation and aggregation
- E
AutoML for automating the feature engineering process
Show answer and explanation
Correct answers: A, B, D
Explanation
To extract and prepare data from Delta Lake for training a generative AI model, you need tools that can filter, transform, and aggregate large datasets efficiently. Databricks SQL supports querying and filtering, Delta Live Tables automate data pipeline management, and the Spark DataFrame API handles transformations and aggregations. MLflow and AutoML, while useful in different stages of the AI workflow, are not directly relevant to the data extraction process.
- A. Correct.
Databricks SQL is well-suited for querying and filtering large datasets stored in Delta Lake, making it an essential tool for efficient data extraction.
- B. Correct.
Delta Live Tables enable the creation and management of reliable, automated data pipelines, which is helpful for applying transformations and preparing the data for the AI model.
- C. Incorrect.
MLflow is primarily used for tracking experiments, models, and parameters, but it is not a tool for data extraction or transformation.
- D. Correct.
Apache Spark DataFrame API provides powerful capabilities for transforming and aggregating large datasets, making it a key tool for this task.
- E. Incorrect.
AutoML is used for automating machine learning tasks such as model selection and hyperparameter tuning, not for data extraction or transformation.