Databricks Generative AI Engineer Associate Question 96
Select 2You are tasked with building a Databricks notebook to retrieve specific customer transaction data from a large dataset stored in Delta Lake. The requirements include filtering transactions from the last 6 months and only retrieving records for 'gold-tier' customers. Which of the following steps should you perform to create the necessary data retrieval tool?
- A
Use Spark SQL to filter the data based on the transaction date and customer tier.
- B
Leverage Delta Lake's time travel feature to retrieve only the latest version of the dataset.
- C
Write a UDF (User-Defined Function) to manually parse and filter the dataset.
- D
Optimize the Delta table to improve query performance before running the data retrieval logic.
- E
Use Databricks Auto Loader to continuously ingest new transaction data before filtering it.
Show answer and explanation
Correct answers: A, D
Explanation
To build an effective data retrieval tool in Databricks, you should use Spark SQL to filter the required data based on the specified conditions (transaction date and customer tier). Optimizing the Delta table ensures that the queries are executed efficiently, which is important when retrieving data from large datasets. Other options, such as time travel or UDFs, do not align with the task requirements and would not be the optimal approach.
- A. Correct.
This is correct because using Spark SQL to filter data based on date and customer tier aligns with the requirements and is an efficient way to retrieve specific subsets of data in Databricks.
- B. Incorrect.
This is incorrect because Delta Lake's time travel feature is used for accessing historical versions of data, not for filtering the latest records based on specific requirements.
- C. Incorrect.
This is incorrect because writing a UDF for filtering is unnecessary when Spark SQL or DataFrame API can efficiently handle the filtering. UDFs should be avoided for tasks that native Spark operations can handle.
- D. Correct.
This is correct because optimizing the Delta table ensures better performance for queries, especially when working with large datasets.
- E. Incorrect.
This is incorrect because Databricks Auto Loader is used for ingesting new data into Delta Lake, which is unrelated to the task of filtering the existing dataset for specific requirements.