Databricks Data Engineer Professional Question 99
Select 3You are tasked with re-designing a data pipeline that processes change data capture (CDC) events in a Delta Lake table. The current pipeline uses a normal Structured Streaming read to process incremental data. You plan to enable Change Data Feed (CDF) on the Delta Lake table and modify the pipeline to handle CDC events. Which of the following steps are required to correctly implement this change?
- A
Enable Change Data Feed (CDF) on the Delta Lake table by setting the 'delta.enableChangeDataFeed' property to 'true'.
- B
Modify the Structured Streaming read to include only the 'insert' and 'delete' events from the '_change_type' column.
- C
Use the 'table_changes' function in a Structured Streaming query to process the CDC output.
- D
Ensure that the Delta Lake table is partitioned before enabling CDF to maintain query performance.
- E
Update downstream logic to handle '_change_type' metadata column values such as 'insert', 'update_preimage', and 'update_postimage'.
Show answer and explanation
Correct answers: A, C, E
Explanation
To process Change Data Capture (CDC) events using Delta Lake's Change Data Feed (CDF), you must first enable CDF on the Delta Lake table by setting the 'delta.enableChangeDataFeed' property to 'true'. The 'table_changes' function is then used in Structured Streaming or batch queries to retrieve the CDC output. Additionally, downstream logic must be updated to interpret the '_change_type' metadata column and handle event types like 'insert', 'update_preimage', 'update_postimage', and 'delete'. Partitioning the table is not a requirement for enabling CDF, and filtering for specific events is performed in downstream processing, not during the read operation.
- A. Correct.
This is correct. Enabling Change Data Feed (CDF) is a mandatory step to capture CDC events from a Delta Lake table. The 'delta.enableChangeDataFeed' property must be set to 'true'.
- B. Incorrect.
This is incorrect. The '_change_type' column is available in the CDC output from CDF, but filtering for specific event types (like 'insert' or 'delete') is not directly done during the streaming read. Instead, it is handled downstream in the processing logic.
- C. Correct.
This is correct. The 'table_changes' function is used to access CDC output in Delta Lake when CDF is enabled. It is the primary function for querying CDC events.
- D. Incorrect.
This is incorrect. Partitioning a Delta Lake table is not a prerequisite for enabling CDF. While partitioning can improve query performance in some cases, it is not required for CDF functionality.
- E. Correct.
This is correct. Downstream logic must be updated to handle the '_change_type' metadata column, which will contain values such as 'insert', 'update_preimage', and 'update_postimage', to properly process the CDC events.