Databricks Data Engineer Professional Question 98
Select 3You are tasked with re-engineering a data pipeline to process change data capture (CDC) events instead of an incremental feed from a normal Structured Streaming read. You enable Change Data Feed (CDF) on a Delta Lake table. Which of the following steps are required to process CDC data in your pipeline?
- A
Enable Change Data Feed (CDF) on the Delta Lake table by setting the appropriate table property.
- B
Use the 'MERGE INTO' operation to directly consume CDC data from the Delta table and apply changes to the target table.
- C
Read the Delta table using the 'readChanges' API to capture CDC events based on a specific version or timestamp.
- D
Filter the '_change_type' column in the CDC output to handle specific event types (e.g., inserts, updates, deletes).
- E
Use a standard Structured Streaming read without any additional configurations to process CDC data.
Show answer and explanation
Correct answers: A, C, D
Explanation
To process CDC data in Delta Lake, you must first enable Change Data Feed (CDF) on the table by setting the table property 'delta.enableChangeDataFeed' to 'true'. Then, you can use the 'readChanges' API to read the CDC output. The '_change_type' column in the CDC output provides metadata about the nature of the changes, which can be filtered to process specific event types like inserts, updates, or deletes. Standard Structured Streaming reads do not support CDC processing, and while 'MERGE INTO' is used for applying changes, it is not used to directly read CDC data.
- A. Correct.
Correct. Enabling Change Data Feed (CDF) on the Delta Lake table is a mandatory step to capture CDC events. This is done by setting the table property 'delta.enableChangeDataFeed' to 'true'.
- B. Incorrect.
Incorrect. While 'MERGE INTO' is used for applying changes, it is not used to directly consume CDC data. The 'readChanges' API is required to read CDC output.
- C. Correct.
Correct. The 'readChanges' API is specifically designed to read CDC events, providing a stream of changes for a specific version or timestamp range.
- D. Correct.
Correct. The '_change_type' column in the CDC output contains metadata about the type of change (e.g., 'insert', 'update_preimage', 'update_postimage', 'delete'). Filtering this column is necessary to handle specific event types appropriately in downstream processing.
- E. Incorrect.
Incorrect. Standard Structured Streaming reads do not capture CDC information. CDC processing requires using specific APIs like 'readChanges' to access change data.