Databricks Data Engineer Professional Question 43
Select 3You are tasked with designing a batch processing job in Databricks to process large volumes of data from a data lake with minimal latency. The data is partitioned by a 'date' column. You want to ensure that the job runs efficiently by leveraging partition pruning and avoiding unnecessary data scans. Which of the following steps should you take when implementing the solution?
- A
Filter the data using the 'date' column in the query.
- B
Enable AQE (Adaptive Query Execution) in the Spark configuration.
- C
Use Delta Lake as the storage format and ensure the 'date' column is defined as a partition column.
- D
Avoid specifying the 'date' column in the WHERE clause to prevent additional overhead.
- E
Persist intermediate DataFrames using the 'MEMORY_AND_DISK' storage level.
Show answer and explanation
Correct answers: A, B, C
Explanation
To optimize a batch processing job that processes large volumes of data from a partitioned data lake, the key steps include leveraging partition pruning and dynamic query optimization. Filtering by the partition column ('date') ensures that only relevant partitions are scanned. Enabling AQE provides query optimization at runtime, and using Delta Lake with a properly defined partition column supports efficient data access. Avoiding the 'date' column in the WHERE clause would defeat the purpose of partition pruning, and while persisting DataFrames may improve other workflows, it is not directly relevant to this specific optimization.
- A. Correct.
Filtering the data using the 'date' column in the query ensures that partition pruning can occur, reducing the amount of data scanned.
- B. Correct.
Enabling AQE (Adaptive Query Execution) allows Spark to optimize the query execution plan dynamically, which can improve the performance of partition pruning and other optimizations.
- C. Correct.
Using Delta Lake and defining the 'date' column as a partition column ensures that the storage system organizes data in a way that supports efficient partition pruning.
- D. Incorrect.
Avoiding the 'date' column in the WHERE clause would prevent partition pruning, leading to full table scans and reduced efficiency. This is not recommended.
- E. Incorrect.
Persisting intermediate DataFrames may help with caching, but it is not directly related to enabling partition pruning or optimizing data scans for batch processing.