Databricks Data Engineer Professional Question 160
Select 3You are tasked with designing a data pipeline in Databricks to process data from a bronze table to a silver table. The requirements are as follows: the pipeline should handle late-arriving data, enforce schema consistency, and remove duplicates based on a composite key ('user_id', 'event_timestamp'). Which of the following steps should you include in your pipeline to meet these requirements?
- A
Read the data incrementally from the bronze table using a watermark on 'event_timestamp' to handle late-arriving data.
- B
Apply schema evolution to allow automatic addition of new columns from the bronze table to the silver table.
- C
Use a window function to identify and remove duplicate rows based on the 'user_id' and 'event_timestamp' composite key.
- D
Filter out records that do not conform to the expected schema using a schema validation step before writing to the silver table.
- E
Write the data directly to the silver table without any transformations to ensure faster processing.
Show answer and explanation
Correct answers: A, C, D
Explanation
To process data from bronze to silver while meeting the requirements, the pipeline must handle late-arriving data, enforce schema consistency, and remove duplicates. Incremental reading with watermarks addresses late-arriving data, schema validation ensures consistency by filtering out invalid records, and deduplication using a window function meets the requirement to remove duplicates based on the composite key. Schema evolution and direct writes without transformations do not align with the stated requirements.
- A. Correct.
Reading the data incrementally with a watermark on 'event_timestamp' ensures the pipeline can process late-arriving data while avoiding processing the same data multiple times.
- B. Incorrect.
Schema evolution is not appropriate here, as the requirement is to enforce schema consistency, not to allow schema drift or automatic column additions.
- C. Correct.
Using a window function is a common approach to deduplicating data based on a composite key, meeting the requirement to remove duplicates.
- D. Correct.
Filtering out records that do not conform to the expected schema ensures that only valid data is written to the silver table, enforcing schema consistency.
- E. Incorrect.
Writing data directly to the silver table without transformations does not meet the requirements for quality enforcement and deduplication.