Databricks Data Engineer Professional Question 107
Select 3You are working on a data pipeline in Databricks using Delta Lake. Your source system generates Change Data Feed (CDF) records, and you need to ensure that delete operations from the source are correctly propagated to your target Delta table. Which of the following steps are necessary to achieve this?
- A
Use the MERGE INTO operation in the target table, matching records using a unique key and handling delete conditions explicitly.
- B
Enable Change Data Feed (CDF) on the target Delta table by setting the 'delta.enableChangeDataFeed' table property.
- C
Query the '_change_type' column from the source Delta table's CDF output to identify deleted rows.
- D
Manually delete rows from the target Delta table without using any automated process.
- E
Ensure the source Delta table has Change Data Feed (CDF) enabled to track changes, including deletes.
Show answer and explanation
Correct answers: A, C, E
Explanation
To propagate deletes using Change Data Feed (CDF), you must first enable CDF on the source Delta table, as it allows tracking of all data changes. The '_change_type' column in the CDF output identifies the type of change, such as 'delete', which can then be handled appropriately. The MERGE INTO operation is a powerful tool that can handle deletes by defining actions explicitly for matched conditions. Enabling CDF on the target table or manually deleting rows are not required or recommended.
- A. Correct.
Correct. The MERGE INTO operation allows you to handle all types of changes (insert, update, and delete) by explicitly defining actions for each condition.
- B. Incorrect.
Incorrect. Enabling CDF on the target table is not required; it is the source Delta table that must have CDF enabled to track changes.
- C. Correct.
Correct. The '_change_type' column in the CDF output of the source Delta table contains metadata about the type of change (e.g., 'delete') and must be used to identify delete operations.
- D. Incorrect.
Incorrect. Manually deleting rows from the target table is not practical or scalable in a data engineering workflow. Automated processes should be used for propagating deletes.
- E. Correct.
Correct. CDF must be enabled on the source Delta table to capture and propagate all data changes, including delete operations.