Databricks Data Engineer Associate Question 403
Select 3A data engineering team is implementing a Change Data Capture (CDC) pipeline using Delta Lake in Databricks. They are using the APPLY CHANGES INTO command to process incremental changes from a source table into a target table. Which of the following statements correctly describe the behavior of APPLY CHANGES INTO?
- A
It automatically handles merging of INSERT, UPDATE, and DELETE operations from the source into the target table.
- B
It requires a unique key column or combination of columns to identify rows to be updated or deleted in the target table.
- C
It performs a full table scan on both the source and target tables during each execution.
- D
It supports advanced configurations like handling late-arriving data and custom update logic.
- E
It can only be used for batch data processing and does not support streaming sources.
Show answer and explanation
Correct answers: A, B, D
Explanation
APPLY CHANGES INTO is a Delta Lake command that simplifies Change Data Capture (CDC) pipelines by efficiently merging incremental changes from a source table into a target table. It automatically handles INSERT, UPDATE, and DELETE operations, requires a unique key for matching rows, and supports advanced configurations like handling late-arriving data. Contrary to common misconceptions, it does not perform full table scans and supports both batch and streaming sources, making it highly versatile for modern data engineering workflows.
- A. Correct.
APPLY CHANGES INTOis designed to handle incremental changes, including INSERT, UPDATE, and DELETE operations automatically. This is a core feature of the command. - B. Correct.
A unique key column or combination of columns is mandatory for
APPLY CHANGES INTOto determine how changes should be applied to the target table. - C. Incorrect.
APPLY CHANGES INTOdoes not perform a full table scan; instead, it processes changes incrementally for efficiency. - D. Correct.
APPLY CHANGES INTOsupports advanced configurations such as custom update logic and handling late-arriving data, making it flexible for various use cases. - E. Incorrect.
APPLY CHANGES INTOsupports both batch and streaming sources, making it suitable for real-time data pipelines as well as batch workloads.