Databricks Data Engineer Professional Question 102
Select 3You are building a data pipeline on Databricks to process change data capture (CDC) data from a Delta Lake table. Previously, the pipeline used incremental reads with Structured Streaming. However, due to a new requirement to track record-level changes (inserts, updates, and deletes), you decide to enable Change Data Feed (CDF) on the Delta table and redesign your processing steps. After enabling CDF on the Delta table, which steps should you implement to correctly process the CDC output?
- A
Use the
table_changesfunction to fetch CDC records and filter by the required version range. - B
Enable CDF by setting the
delta.enableChangeDataFeedproperty to true for the Delta table. - C
Use the
_change_typecolumn in the CDC output to handle inserts, updates, and deletes appropriately. - D
Continue using the default Structured Streaming read without any modifications.
- E
Set up a Delta Live Table to automatically process CDC changes without enabling CDF explicitly.
Show answer and explanation
Correct answers: A, B, C
Explanation
To process change data capture (CDC) data from a Delta Lake table, you must enable Change Data Feed (CDF) by setting the delta.enableChangeDataFeed property on the table. Once CDF is enabled, you can use the table_changes function to fetch CDC records for a specific version range. The _change_type column in the CDC output provides the type of change (insert, update, delete), which is critical to handle the data correctly. The default Structured Streaming read does not support CDC, and additional configuration is required to ensure the pipeline processes CDC output.
- A. Correct.
Correct: The
table_changesfunction is a key feature of Delta Lake with CDF enabled and must be used to fetch the CDC output for a given version range. - B. Correct.
Correct: Enabling CDF requires setting the
delta.enableChangeDataFeedproperty to true at the table level before CDC data can be generated. - C. Correct.
Correct: The
_change_typecolumn in the CDC output is essential to distinguish between inserts, updates, and deletes, and is a critical part of processing CDC data. - D. Incorrect.
Incorrect: The default Structured Streaming read does not process CDC data with record-level change information. You need to use the CDF features to explicitly handle these changes.
- E. Incorrect.
Incorrect: Delta Live Tables can process Delta tables, but enabling CDF explicitly is still required to capture record-level changes for your pipeline.