Databricks Data Engineer Professional Question 105
Select 3You are managing a Delta Lake table in Databricks that tracks customer orders. The table is configured with Change Data Feed (CDF) enabled. Your team is building a downstream application that requires both new and updated records while also ensuring that deleted records are processed and removed correctly from the downstream system. Which of the following steps should you take to propagate the deletes to the downstream system using CDF?
- A
Query the Change Data Feed using the
readChangeFeedAPI and filter for rows where the_change_typecolumn is 'delete'. - B
Update the downstream target system by removing rows matching the primary key of records with
_change_typeas 'delete'. - C
Query the Delta table directly using the
readAPI and filter for rows where the_change_typecolumn is 'delete'. - D
Ensure that the downstream system supports processing of delete operations based on primary keys.
- E
Write the delete records back to the Delta Lake table to mark them as processed.
Show answer and explanation
Correct answers: A, B, D
Explanation
To propagate deletes using Delta Lake's Change Data Feed (CDF), you should query the table using the readChangeFeed API to filter rows where _change_type is 'delete'. These rows should then be processed by removing them from the downstream system based on their primary keys. Additionally, the downstream system must support handling deletes to ensure proper propagation.
- A. Correct.
Correct. The
readChangeFeedAPI is the correct method to query the Change Data Feed and filter for deletes using the_change_typecolumn. - B. Correct.
Correct. To propagate deletes, the downstream system needs to remove records matching the primary key of deleted rows.
- C. Incorrect.
Incorrect. The
readAPI does not provide access to change data feed-specific metadata like_change_type. This makes it unsuitable for identifying deletes. - D. Correct.
Correct. The downstream system must handle deletions properly by processing the primary keys of deleted records.
- E. Incorrect.
Incorrect. Writing delete records back to the Delta Lake table is unnecessary and does not propagate deletes to downstream systems.