Databricks Data Engineer Professional Question 168
Select 2A data engineering team is designing a Delta Lake pipeline to enforce data quality for a large transactional dataset. They need to ensure that incoming records comply with specific schema requirements and business rules, such as ensuring all transaction amounts are positive and that the transaction IDs are unique. Which of the following approaches should they implement to enforce data quality effectively while optimizing for performance and reliability?
- A
Use Delta Lake constraints like NOT NULL and CHECK constraints to enforce basic schema rules during data writes.
- B
Apply data validation logic in an upstream application layer before writing the data to Delta Lake.
- C
Use Delta Live Tables (DLT) expectations to define and enforce data quality rules at each stage of the pipeline.
- D
Enable Delta Lake's auto-compaction feature to validate data quality during file merging.
- E
Create a custom job to periodically scan the Delta Lake table and remove records that violate data quality rules.
Show answer and explanation
Correct answers: A, C
Explanation
To enforce data quality in Delta Lake, it is best to use built-in features such as constraints (e.g., NOT NULL, CHECK) and managed frameworks like Delta Live Tables (DLT) to define expectations at each stage of the pipeline. These approaches are efficient, reliable, and ensure data quality is enforced as part of the data ingestion and transformation process. Upstream validation and reactive clean-up processes are less reliable or efficient, and auto-compaction does not address data quality enforcement at all.
- A. Correct.
Delta Lake supports constraints such as NOT NULL and CHECK constraints, which can enforce schema-level data quality rules directly during data writes. This approach is effective for basic validations and ensures that invalid data does not enter the table.
- B. Incorrect.
While upstream validation can help reduce bad data ingestion, relying solely on this approach introduces risks because the Delta Lake itself will not enforce these rules. Data quality should ideally also be enforced closer to the data storage layer.
- C. Correct.
Delta Live Tables (DLT) is a managed framework in Databricks that allows you to define and enforce data quality rules through expectations. These expectations can be applied at multiple stages of the pipeline, making it a robust and flexible option for enforcing business rules and data quality.
- D. Incorrect.
Delta Lake's auto-compaction feature is designed for file optimization and does not enforce data quality rules. It is unrelated to data validation or constraints.
- E. Incorrect.
Creating a custom job to scan and clean data is reactive and inefficient compared to enforcing data quality rules during data ingestion or transformation. This approach may increase operational complexity and delay issue resolution.