Databricks Data Engineer Professional Question 106
Select 3You are a data engineer working on a Delta Lake table that logs user activity. The table is being used downstream for analytics and machine learning. Recently, your organization implemented a new policy requiring the deletion of user data when users opt out, and you must ensure these deletions propagate correctly to downstream systems. You decide to leverage Delta Change Data Feed (CDF). Which of the following steps will ensure that deletes are reflected downstream?
- A
Enable Change Data Feed (CDF) on the Delta table by setting the appropriate table property.
- B
Query the
_change_typecolumn from the Delta CDF output to identify delete records. - C
Use the
MERGEstatement to update the downstream table with the changes from the CDF output. - D
Use the
TRUNCATEcommand to delete all records from the downstream table and reload data from the source. - E
Enable Delta Lake's VACUUM operation before propagating changes to downstream tables.
Show answer and explanation
Correct answers: A, B, C
Explanation
To propagate deletes using Delta Change Data Feed (CDF), you must first enable CDF on the Delta table. Then, by querying the CDF output, you can identify delete operations through the _change_type column. Finally, you can use the MERGE statement to apply these changes to downstream systems efficiently. This ensures that the deletions are reflected downstream while maintaining the integrity of the data pipeline.
- A. Correct.
Correct: To use Change Data Feed (CDF), it must first be enabled on the Delta table by setting the table property
delta.enableChangeDataFeed = true. This ensures the table starts tracking changes, including deletes. - B. Correct.
Correct: The
_change_typecolumn in the CDF output is crucial for identifying the type of change (e.g., 'delete', 'insert', 'update_postimage'). This allows you to specifically detect and handle delete records. - C. Correct.
Correct: The
MERGEstatement can be used to apply the changes from the CDF output to downstream tables, ensuring deletes and other changes are properly propagated. - D. Incorrect.
Incorrect: Using
TRUNCATEto delete all records and reload the entire dataset is inefficient and does not leverage the incremental nature of CDF. This approach goes against best practices for using Delta Lake. - E. Incorrect.
Incorrect: VACUUM is used to remove old files and is unrelated to propagating changes with CDF. Enabling VACUUM without proper retention settings could even risk losing necessary historical data for propagation.