Databricks Data Engineer Professional Question 175
Single answerA data engineering team is working with Delta Lake tables and wants to ensure that only valid data is written to a table storing customer information. Specifically, they want to enforce that the 'email' column cannot contain NULL values and the 'age' column must be greater than or equal to 18. Which approach should the team use to implement these constraints?
- A
Use Delta Lake table constraints by defining a CHECK constraint during table creation to enforce these rules.
- B
Create a Delta Lake table with AUTO OPTIMIZE enabled and rely on Delta Lake's automatic validation of data.
- C
Write a Spark job to validate the data before writing it into the Delta Lake table.
- D
Enable Delta Lake's data quality enforcement feature, which automatically blocks writes with invalid data.
Show answer and explanation
Correct answer: A
Explanation
Delta Lake provides built-in support for constraints, such as NOT NULL and CHECK constraints, to prevent invalid data from being written to tables. These constraints can be specified during table creation or later using SQL commands like ALTER TABLE. This declarative approach simplifies data validation and ensures data integrity at the storage level. Other options either misrepresent Delta Lake's features or require more manual effort than necessary.
- A. Correct.
This is the correct option. Delta Lake supports table constraints, like NOT NULL and CHECK constraints, which can be defined during table creation or alteration to enforce data integrity rules.
- B. Incorrect.
This is incorrect. AUTO OPTIMIZE is used to optimize file management within Delta Lake, but it does not validate data or enforce constraints.
- C. Incorrect.
This is partially valid but not the best approach. While a Spark job can validate data before writing, it requires additional coding and logic, whereas Delta Lake's built-in constraints provide a simpler, declarative solution.
- D. Incorrect.
This is incorrect. Delta Lake does not have a generic 'data quality enforcement feature' that automatically validates data. Constraints like CHECK and NOT NULL must be explicitly defined.