Databricks Data Engineer Associate Question 407
Single answerYou are implementing a Change Data Capture (CDC) pipeline using Delta Lake in Databricks. You use the APPLY CHANGES INTO statement to merge CDC data from a source table into a target table. Which of the following statements is true about the behavior of APPLY CHANGES INTO?
- A
It automatically handles inserts, updates, and deletes based on the primary key defined in the target table.
- B
It requires you to explicitly define how to handle inserts, updates, and deletes in the query.
- C
It processes changes in the source table incrementally, based on a watermark column.
- D
It requires the target table schema to exactly match the source table schema before merging.
Show answer and explanation
Correct answer: C
Explanation
The APPLY CHANGES INTO statement in Databricks is designed to handle Change Data Capture (CDC) scenarios efficiently. It operates incrementally by processing data based on a watermark column, ensuring only new or updated records are processed. This simplifies the implementation of CDC pipelines by automatically managing inserts, updates, and deletes without requiring explicit instructions from the user. The schema of the target table can also accommodate schema evolution, making APPLY CHANGES INTO highly flexible for real-world use cases.
- A. Incorrect.
Incorrect: APPLY CHANGES INTO automatically handles inserts, updates, and deletes, but it relies on the primary key defined in the source table, not the target table.
- B. Incorrect.
Incorrect: APPLY CHANGES INTO does not require explicit definitions for handling inserts, updates, and deletes. It is designed to simplify the process by automatically managing these operations.
- C. Correct.
Correct: APPLY CHANGES INTO processes CDC data incrementally by tracking changes based on a specified watermark column in the source table.
- D. Incorrect.
Incorrect: The schema of the target table does not need to exactly match the schema of the source table. APPLY CHANGES INTO can handle schema evolution if configured properly.