Databricks Data Engineer Professional Question 100
Select 3You are working on a Delta Lake-based data pipeline where you need to process Change Data Capture (CDC) events instead of an incremental feed from a standard Structured Streaming read. You have a Delta table named 'sales_data' and want to enable Change Data Feed (CDF) to track and process changes (inserts, updates, deletions). Which of the following steps are necessary to achieve this and redesign your processing pipeline?
- A
Enable Change Data Feed (CDF) on the 'sales_data' table by setting the Delta table property.
- B
Use the
readStream().load()method to read the Delta table and filter for_change_typecolumn. - C
Use the
table_changes()function to read CDC data from the Delta table and process inserts, updates, and deletes. - D
Configure the Delta table's schema to include a
_change_typecolumn before enabling CDF. - E
Set the
readChangeFeedoption totruewhen reading the Delta table to process CDC events.
Show answer and explanation
Correct answers: A, C, E
Explanation
To process CDC output from a Delta table with Change Data Feed (CDF) enabled, you must first enable CDF on the table by setting the appropriate table property. Then, you can leverage the table_changes() function to read CDC data or use the readChangeFeed option when reading the Delta table in a streaming pipeline. This approach allows you to redesign your data processing steps to handle inserts, updates, and deletions effectively.
- A. Correct.
Enabling Change Data Feed (CDF) on a Delta table is a prerequisite to track and process CDC events. This is a required step.
- B. Incorrect.
The
readStream().load()method does not provide access to the_change_typecolumn. You need to use thetable_changes()function or enablereadChangeFeedto process CDC data. - C. Correct.
The
table_changes()function is specifically designed to read the CDC data from a Delta table and is essential for processing inserts, updates, and deletions. - D. Incorrect.
You do not need to manually configure a
_change_typecolumn in the Delta table schema. This column is automatically available when CDF is enabled. - E. Correct.
The
readChangeFeedoption is required when using Structured Streaming to process CDC events from a Delta table. It allows you to access changes such as inserts, updates, and deletes.