Databricks Data Engineer Professional Question 177
Select 2You are tasked with ensuring data quality in a Delta Lake table named 'orders'. Specifically, you want to enforce that the 'order_date' column cannot have future dates and the 'order_amount' column cannot have negative values. What is the appropriate approach to achieve this?
- A
Use Delta Lake's
CHECKconstraints to enforce both conditions during writes. - B
Implement a data validation step in your ETL pipeline before writing to the Delta Lake table.
- C
Modify the Delta table schema to include constraints using SQL ALTER TABLE with
ADD CONSTRAINT. - D
Enable Delta Lake's automatic constraint enforcement feature through table properties.
- E
Leverage Delta's transaction log to manually track and reject invalid records.
Show answer and explanation
Correct answers: A, C
Explanation
To prevent bad data from being written to Delta Lake tables, you should use Delta Lake's CHECK constraints or modify the table schema with SQL to add constraints directly. These approaches ensure that invalid data, such as future dates or negative values, is rejected during write operations. While other methods like ETL validation can complement this, they are not substitutes for Delta Lake's built-in constraint mechanisms.
- A. Correct.
Delta Lake supports the use of
CHECKconstraints, which can validate data during writes to ensure conditions like no future dates for 'order_date' and no negative values for 'order_amount'. - B. Incorrect.
While adding a validation step in the ETL pipeline can help, it does not natively leverage Delta Lake's constraint features and could miss some invalid data if the pipeline is bypassed.
- C. Correct.
Delta Lake allows you to modify the table schema with SQL commands like
ALTER TABLEto add constraints, enforcing rules such as no future dates or negative values. - D. Incorrect.
Delta Lake does not have an automatic constraint enforcement feature that can be enabled through table properties. Constraints must be explicitly defined.
- E. Incorrect.
Manually tracking invalid records in the transaction log is not a recommended or practical approach for enforcing constraints in Delta Lake.