Databricks Data Engineer Professional Question 97
Select 3A company is using a Delta Lake table to manage its sales data and wants to enable Change Data Feed (CDF) to capture changes in the table. Previously, they used normal Structured Streaming reads to process incremental updates. However, they now want to redesign their pipeline to process the CDC output provided by CDF. Which of the following steps are required to implement the new design?
- A
Enable Change Data Feed (CDF) on the Delta table by setting the appropriate table property.
- B
Modify the Structured Streaming query to read from the table's '_change_data' path to capture CDC output.
- C
Manually create a schema to handle the '_change_type' column in the CDC data.
- D
Filter the '_change_type' column in the CDC output to process only relevant change types such as 'insert' or 'update_postimage'.
- E
Disable the 'mergeSchema' option in the read operation to prevent schema evolution issues.
Show answer and explanation
Correct answers: A, B, D
Explanation
To redesign the data pipeline to process CDC output from Delta Lake tables, you must first enable CDF by setting the appropriate Delta table property. The Structured Streaming query should then read from the '_change_data' path to access the CDC output. The '_change_type' column in the CDC data provides information about the type of change, and filtering this column ensures only relevant changes are processed. Manually creating a schema or disabling the 'mergeSchema' option is not required for implementing this design.
- A. Correct.
Correct. Enabling Change Data Feed (CDF) requires setting the 'delta.enableChangeDataFeed' property to 'true' on the Delta table.
- B. Correct.
Correct. To process CDC data, the read operation must point to the '_change_data' path of the Delta table, which exposes the CDC output.
- C. Incorrect.
Incorrect. The '_change_type' column is automatically included in the CDC output schema, so creating a schema manually is not required.
- D. Correct.
Correct. The '_change_type' column in the CDC output provides information about the type of change (e.g., 'insert', 'update_preimage', 'update_postimage', 'delete'). Filtering this column helps process relevant changes effectively.
- E. Incorrect.
Incorrect. Disabling the 'mergeSchema' option is unrelated to processing CDC data and could prevent handling schema evolution during future updates.