Databricks Data Engineer Associate Question 222
Single answerYou are tasked with designing an incremental data pipeline using Databricks to process data arriving in a Delta Lake table. The data arrives in multiple batches throughout the day, and downstream consumers need access to the latest processed data as soon as it is available. Which of the following approaches should you use to ensure efficient and reliable incremental processing?
- A
Use structured streaming to continuously process new data from the Delta table and write the results to a new Delta table.
- B
Run a full table scan on the Delta table every time new data arrives and overwrite the downstream table.
- C
Manually track processed rows using an external file and filter out already processed data during each batch.
- D
Use the Delta Lake 'MERGE INTO' operation to process new data incrementally and update the downstream table.
Show answer and explanation
Correct answer: A
Explanation
Structured streaming with Delta Lake is the optimal solution for incremental data processing in Databricks. It allows for continuous processing of new data with minimal latency, making it well-suited for scenarios where data arrives in batches and downstream consumers require timely access to updates.
- A. Correct.
This is the correct approach because structured streaming in Databricks can efficiently process new data incrementally when paired with Delta Lake, ensuring low-latency updates for downstream consumers.
- B. Incorrect.
This approach is inefficient as it requires scanning the entire table for each batch, leading to unnecessary computation and resource usage.
- C. Incorrect.
Manually tracking processed rows is prone to errors and is not a reliable or scalable method for incremental processing in Databricks.
- D. Incorrect.
While 'MERGE INTO' is useful for certain update operations, it is not suitable for an ongoing incremental data pipeline where structured streaming is the recommended approach.