Databricks Data Engineer Professional Question 157
Select 4You are tasked with designing a data pipeline in Databricks to process data from a bronze table to a silver table. The pipeline must handle incremental updates, enforce data quality by filtering out invalid records, and deduplicate data based on a unique identifier column called 'transaction_id'. Which steps should you include in your pipeline?
- A
Use a watermark on the 'timestamp' column to process only new data.
- B
Apply validation rules to filter out rows with null values in critical columns.
- C
Use the 'dropDuplicates' function, specifying 'transaction_id' as the unique identifier.
- D
Overwrite the silver table with the entire dataset from the bronze table on every run.
- E
Use a Delta Lake MERGE operation to upsert data into the silver table.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To process data from a bronze to a silver table in a Databricks pipeline, you must ensure incremental processing, enforce data quality, and handle deduplication. Incremental processing can be achieved using watermarks, while data quality enforcement is done by applying validation rules. Deduplication is handled using functions like 'dropDuplicates' on a unique identifier such as 'transaction_id'. Finally, using Delta Lake MERGE ensures efficient upserts into the silver table, aligning with best practices for Delta Lake pipelines.
- A. Correct.
Using a watermark on the 'timestamp' column ensures that only new data is processed incrementally, which is a best practice for handling streaming or incremental processing.
- B. Correct.
Applying validation rules like filtering out rows with nulls in critical columns ensures data quality enforcement, which is a key requirement when moving data from bronze (raw) to silver (cleaned).
- C. Correct.
The 'dropDuplicates' function is essential for deduplication when working with datasets that may contain duplicates, ensuring only unique rows based on 'transaction_id' are written to the silver table.
- D. Incorrect.
Overwriting the silver table with the entire dataset on every run is inefficient and contradicts the incremental processing requirement.
- E. Correct.
Using a Delta Lake MERGE operation to upsert data into the silver table allows for efficient handling of both new records and updates during the transformation process.