Databricks Data Engineer Associate Question 218
Single answerYou are working on a Databricks project where you need to process incremental updates from a Delta table named 'customer_transactions'. The goal is to capture only the new rows added since the last processing and write them to another Delta table named 'processed_transactions'. Which of the following approaches ensures efficient incremental processing?
- A
Use the Delta Lake
MERGEoperation to identify and process only the new rows. - B
Use the Delta Lake
OPTIMIZEcommand before reading the data. - C
Use the
readStreammethod withreadChangeFeedenabled to capture the new rows. - D
Filter the Delta table using the
_commit_versionmetadata column to identify new rows.
Show answer and explanation
Correct answer: C
Explanation
To process incremental updates efficiently in Databricks, the recommended approach is to use Delta Lake's Change Data Feed (CDF) functionality. Enabling the readChangeFeed option while performing streaming operations allows you to capture only the new rows or changes made to the Delta table since the last checkpoint. This avoids the need to process the entire dataset repeatedly and ensures efficient incremental data processing.
- A. Incorrect.
This is incorrect because the
MERGEoperation is used for upserts and merging datasets, not for capturing incremental updates efficiently. - B. Incorrect.
This is incorrect because the
OPTIMIZEcommand is used to compact files for performance improvements, not for capturing incremental changes. - C. Correct.
This is correct because enabling the
readChangeFeedoption withreadStreamallows the system to capture new rows (and other changes) incrementally from the Delta table. - D. Incorrect.
This is incorrect because the
_commit_versionmetadata is not a recommended or supported way to filter for incremental changes in Delta Lake.