Databricks Generative AI Engineer Associate Question 91
Single answerYou are tasked with building a pipeline to retrieve customer transaction data from a large-scale data lake for use in training a generative AI model. The data is stored in Delta Lake format, and the users of the pipeline require specific transaction records filtered by date and customer ID. Which of the following tools or approaches would be most appropriate to ensure efficient data retrieval and scalability?
- A
Use Databricks SQL to query the Delta Lake data with filters for date and customer ID.
- B
Export the entire Delta Lake data to a CSV file and then filter it using Pandas in a Python script.
- C
Use Databricks Auto Loader to stream the Delta Lake data into memory and apply filters in real-time.
- D
Create a Spark SQL query in a Databricks notebook to filter the Delta Lake data by date and customer ID.
Show answer and explanation
Correct answer: A
Explanation
Databricks SQL is the most appropriate tool for this scenario because it is specifically designed for querying and analyzing large-scale Delta Lake data. It provides efficient filtering and a scalable infrastructure, making it suitable for extracting the required customer transaction records by date and customer ID.
- A. Correct.
Databricks SQL is optimized for querying large-scale data stored in Delta Lake. It supports serverless infrastructure, efficient filtering, and scalability for production-grade pipelines.
- B. Incorrect.
Exporting the entire dataset to a CSV file and filtering it with Pandas is inefficient and impractical for large-scale data, as it requires significant memory and processing time.
- C. Incorrect.
Databricks Auto Loader is designed for streaming new data into Delta Lake, not for querying and filtering existing data. It is not the correct approach for this scenario.
- D. Incorrect.
While Spark SQL in a Databricks notebook can filter Delta Lake data, it is less efficient and not as scalable as using Databricks SQL for this specific task.