Databricks Data Engineer Professional Question 164
Select 2A data engineering team is working on a Delta Lake table to store customer transactions. They need to enforce data quality rules such as ensuring that transaction amounts are non-negative and each record has a valid customer ID. Which approach should they use to enforce these constraints, and why?
- A
Use Delta Lake's built-in CHECK constraints to enforce rules like non-negative transaction amounts.
- B
Implement data quality checks using Delta Live Tables' expectations to validate data during ingestion.
- C
Write custom PySpark transformations to filter out invalid rows before writing to Delta Lake.
- D
Use Databricks SQL to periodically query the table and identify records that violate the constraints.
- E
Enable Delta Lake's schema enforcement to automatically reject records with invalid fields.
Show answer and explanation
Correct answers: A, B
Explanation
Delta Lake's CHECK constraints and Delta Live Tables' expectations are proactive and declarative approaches to enforce data quality rules. CHECK constraints enforce column-level rules at the storage level, ensuring data integrity. Delta Live Tables' expectations validate data during ingestion, allowing for flexibility and early prevention of bad data. These approaches are efficient and reduce the need for manual intervention or reactive solutions.
- A. Correct.
Delta Lake's built-in CHECK constraints allow you to enforce column-level rules, such as ensuring transaction amounts are non-negative. This is a strong choice for column-specific data quality enforcement.
- B. Correct.
Delta Live Tables provides a declarative way to define and enforce expectations on data during ingestion, making it well-suited for validating records like customer IDs before they are written to the Delta Lake table.
- C. Incorrect.
While custom PySpark transformations can be used to filter data, this approach requires additional code maintenance and does not inherently enforce the rules at the Delta Lake level.
- D. Incorrect.
Using Databricks SQL to identify invalid records is a reactive approach and does not prevent bad data from being written into the Delta Lake table.
- E. Incorrect.
Schema enforcement in Delta Lake ensures data adheres to the defined schema (e.g., data types or column presence) but does not support custom constraints like checking for non-negative values or valid IDs.