Databricks Data Engineer Professional Question 159
Select 4You are tasked with transforming a streaming dataset from the bronze layer to the silver layer in Databricks. The dataset contains duplicate records and some invalid rows that do not meet the schema requirements. Which of the following steps should you implement to ensure incremental processing, quality enforcement, and deduplication during the transformation?
- A
Use a watermark on the streaming query to handle late-arriving data during incremental processing.
- B
Apply a schema enforcement mechanism to filter out records that do not meet the schema requirements.
- C
Use the
dropDuplicates()function on the primary key columns to remove duplicate records. - D
Persist the transformed data directly into the gold layer, skipping the silver layer.
- E
Use a checkpoint to maintain state and ensure exactly-once processing for the streaming query.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To process data from the bronze to silver layer in Databricks, it is important to implement incremental processing, quality enforcement, and deduplication. Incremental processing is achieved using watermarks and checkpoints, ensuring that the system handles late-arriving data and maintains state. Schema enforcement is applied to filter out invalid records, and deduplication is performed to remove duplicate records based on primary key columns. Skipping the silver layer is not recommended as it is a critical step in the Lakehouse architecture for intermediate data cleansing and preparation before moving to the gold layer.
- A. Correct.
Using a watermark is essential for handling late-arriving data during incremental processing, ensuring that only relevant data within the defined time window is processed.
- B. Correct.
Schema enforcement ensures that invalid records that do not meet the schema requirements are filtered out, maintaining data quality.
- C. Correct.
The
dropDuplicates()function is used to remove duplicate records based on specified primary key columns, ensuring data consistency in the silver layer. - D. Incorrect.
Persisting data directly into the gold layer skips the silver layer, which contradicts the principle of layered architecture in the Lakehouse design. The silver layer is necessary for intermediate cleansing and deduplication.
- E. Correct.
Using a checkpoint is crucial for maintaining state and enabling exactly-once processing in structured streaming, which ensures reliable incremental processing.