Databricks Data Engineer Professional Question 161
Select 3You are tasked with designing a data pipeline in Databricks to process IoT sensor data from a bronze table to a silver table. The incoming data in the bronze table might contain duplicate records and null values in critical columns. Your pipeline must ensure incremental processing, enforce data quality by dropping records with null values in critical columns, and remove duplicates based on a unique 'sensor_id' and 'event_timestamp' combination. Which of the following steps should you include in your pipeline?
- A
Use a watermark on the stream to ensure late data is managed during incremental processing.
- B
Filter out records with null values in critical columns using a WHERE clause or a filter transformation.
- C
Perform a dropDuplicates operation on the 'sensor_id' and 'event_timestamp' columns to remove duplicates.
- D
Write the results directly to the gold table without creating a silver table to simplify processing.
- E
Implement a merge operation to ensure new records are incrementally added to the silver table.
Show answer and explanation
Correct answers: B, C, E
Explanation
To process data from the bronze table to the silver table with incremental processing, quality enforcement, and deduplication, the pipeline must include filtering out null values in critical columns, deduplication based on a unique key, and incremental updates using a merge operation. These steps meet the specified requirements while ensuring the integrity and quality of the processed data.
- A. Incorrect.
Using a watermark on the stream is helpful for handling late-arriving data, but it is not directly required for deduplication or quality enforcement in this scenario.
- B. Correct.
Filtering out records with null values in critical columns is essential for enforcing data quality as per the requirements.
- C. Correct.
Performing a dropDuplicates operation on the 'sensor_id' and 'event_timestamp' columns is necessary to remove duplicates based on the specified unique combination.
- D. Incorrect.
Writing results directly to the gold table would bypass the silver table, which is against the pipeline design requirement of processing data from bronze to silver.
- E. Correct.
A merge operation ensures that new records are incrementally updated in the silver table, aligning with the requirement for incremental processing.