Databricks Data Engineer Professional Question 162
Select 3You are tasked with designing a data pipeline in Databricks to process data from a bronze table into a silver table. The bronze table contains raw data with potential duplicates and missing values. The silver table must only contain deduplicated data and enforce schema consistency. Which steps should you include in your pipeline?
- A
Use a watermark to filter out late-arriving data during the read operation from the bronze table.
- B
Apply a DISTINCT operation or a window function with row_number() to remove duplicate records.
- C
Use a Delta MERGE operation to enforce schema and upsert records into the silver table.
- D
Write directly to the silver table from the bronze table without applying any transformations.
- E
Implement schema validation to ensure the incoming data matches the expected schema before writing to the silver table.
Show answer and explanation
Correct answers: B, C, E
Explanation
To process data from a bronze table to a silver table, you need to ensure data quality by deduplicating records, enforcing schema validation, and maintaining consistency. Deduplication can be achieved using DISTINCT or window functions, schema consistency can be enforced with Delta MERGE or schema validation, and transformations should always be applied before writing to the silver table.
- A. Incorrect.
Using a watermark is typically applied during streaming operations to handle late-arriving data but is not directly relevant to deduplication or schema consistency in this scenario.
- B. Correct.
This is correct. Deduplication can be achieved using DISTINCT or a window function with row_number(), ensuring only unique records are processed into the silver table.
- C. Correct.
This is correct. A Delta MERGE operation can enforce schema evolution and handle upserts, ensuring that only valid records are written to the silver table.
- D. Incorrect.
This is incorrect. Writing directly from the bronze table without applying transformations would not handle deduplication or schema enforcement, resulting in poor-quality data in the silver table.
- E. Correct.
This is correct. Schema validation ensures that records conform to the expected schema before they are written to the silver table, maintaining data quality.