Databricks Data Engineer Professional Question 158
Select 3You are tasked with designing a pipeline in Databricks to process raw data from a bronze table into a clean, deduplicated silver table. The bronze table contains raw IoT sensor data with potential duplicates. The following steps are provided to you:
- Read the raw data from the bronze table incrementally.
- Deduplicate the data based on 'sensor_id' and 'timestamp'.
- Apply basic quality checks to filter out rows with null 'sensor_id' or invalid 'temperature' readings (e.g., below -50 or above 150).
- Write the clean, deduplicated data into the silver table in append mode.
Which of the following operations should you include in your pipeline to meet the requirements?
- A
Use the
dropDuplicatesmethod on 'sensor_id' and 'timestamp' columns to remove duplicates. - B
Filter the data using a condition to exclude rows with null values in 'sensor_id' or temperature readings outside the valid range.
- C
Write the data to the silver table using overwrite mode to ensure no duplicate records exist.
- D
Use the
mergeoperation to handle incremental updates from the bronze table to the silver table. - E
Read the bronze table using structured streaming to enable incremental processing.
Show answer and explanation
Correct answers: A, B, E
Explanation
To design a pipeline for processing data from a bronze table to a silver table, incremental processing, deduplication, and quality enforcement are key. Reading the data incrementally with structured streaming enables efficient processing of new data. Deduplication ensures no duplicate records are carried over, and quality checks filter out invalid or incomplete rows. Writing the data in append mode ensures new clean data is continually added to the silver table without overwriting existing records.
- A. Correct.
This is correct because the
dropDuplicatesmethod is a standard approach to deduplicate data based on specific columns like 'sensor_id' and 'timestamp'. It ensures unique records are written to the silver table. - B. Correct.
This is correct because applying quality checks is essential to enforce data integrity, such as filtering out rows with null 'sensor_id' or invalid 'temperature' readings.
- C. Incorrect.
This is incorrect because writing the data in overwrite mode would replace all existing data in the silver table, which is not suitable for incremental processing.
- D. Incorrect.
This is incorrect because the
mergeoperation is typically used for slowly changing dimensions (SCDs) or upsert operations, but it is not required for this use case where deduplication and filtering are the main requirements. - E. Correct.
This is correct because reading the bronze table using structured streaming allows the pipeline to process data incrementally, meeting the requirement for incremental processing.