Databricks Data Engineer Professional Question 101
Select 3You are working on a Delta Lake table that receives frequent updates and deletions. Previously, your data pipeline processed incremental data using a normal Structured Streaming read. To optimize your pipeline, you want to enable Change Data Feed (CDF) on the table and redesign your data processing steps to handle CDC (Change Data Capture) output. Which of the following steps should you take to achieve this?
- A
Enable Change Data Feed (CDF) on the Delta Lake table by setting the table property
delta.enableChangeDataFeedtotrue. - B
Use the
input_file_name()function to identify changed rows in the Delta table. - C
Replace your Structured Streaming read with a query on the
_change_typecolumn available in the CDF output to identify inserted, updated, and deleted rows. - D
Query the CDC output using the
table_changesfunction with a specific start and end version or timestamp. - E
Disable schema evolution on the Delta table to prevent schema conflicts when processing CDC data.
Show answer and explanation
Correct answers: A, C, D
Explanation
To redesign your data pipeline to process CDC output after enabling Change Data Feed (CDF) on a Delta Lake table, you must enable CDF by setting the appropriate table property, query the CDC data using the table_changes function, and use the _change_type column to identify the type of changes (insert, update, delete). Other options, such as using input_file_name() or disabling schema evolution, are unrelated to CDC processing and do not contribute to solving the problem.
- A. Correct.
This is correct. To enable Change Data Feed on a Delta Lake table, the table property
delta.enableChangeDataFeedmust be set totrueeither during table creation or by altering the table properties. - B. Incorrect.
This is incorrect. The
input_file_name()function identifies the file origin of a row but does not directly support CDC processing or detect changes like inserts, updates, and deletes. - C. Correct.
This is correct. After enabling CDF, the
_change_typecolumn in the CDF output can be used to filter rows based on their change type (e.g., inserts, updates, deletes). - D. Correct.
This is correct. The
table_changesfunction allows you to query the CDC output for a range of versions or timestamps to fetch only the changed data. - E. Incorrect.
This is incorrect. Disabling schema evolution is not related to processing CDC data and could introduce limitations when dealing with schema changes in your pipeline.