Databricks Data Engineer Professional Question 108
Select 3A company maintains a Delta Lake table that stores customer data. Due to GDPR regulations, customers can request their data to be deleted. You have been tasked with ensuring that these deletions are propagated to downstream systems using Change Data Feed (CDF). Which of the following steps must you perform to correctly propagate the deletes?
- A
Enable Change Data Feed on the Delta table by setting the appropriate table property.
- B
Use the
MERGEstatement in downstream systems to process delete records based on the_change_typecolumn. - C
Filter the CDF output for records where
_change_typeis set to 'delete'. - D
Run the
VACUUMcommand on the Delta table before querying the Change Data Feed. - E
Use the
DELETEstatement directly on the downstream system without referencing the CDF output.
Show answer and explanation
Correct answers: A, B, C
Explanation
To propagate deletes using Change Data Feed (CDF), the feature must first be enabled on the Delta table. The CDF output provides a _change_type column that indicates the type of operation (e.g., insert, update, delete). Filtering for 'delete' in the _change_type column allows identifying records to delete, which can then be applied in downstream systems using a MERGE statement or similar logic. Steps unrelated to CDF, such as running VACUUM or directly issuing DELETE without referencing CDF, will not achieve the desired outcome.
- A. Correct.
Change Data Feed must be enabled on the Delta table by setting the
delta.enableChangeDataFeedproperty totrue. Without this, no change data (including deletes) will be captured. - B. Correct.
The
MERGEstatement is commonly used to apply both insert, update, and delete operations in downstream systems. By referencing the_change_typecolumn in the CDF output, deletes can be correctly applied downstream. - C. Correct.
Filtering the CDF output for records where
_change_typeis 'delete' enables you to isolate only the delete operations. This is necessary to ensure that deletions are propagated properly. - D. Incorrect.
The
VACUUMcommand is used to clean up old files and has no relevance to querying data from the Change Data Feed. Using it here would not help propagate deletes. - E. Incorrect.
The
DELETEstatement in downstream systems requires specific information about which records to delete. Without referencing the CDF output, this method cannot propagate deletes accurately.