Databricks Data Engineer Professional Question 163
Single answerA data engineering team is working on a Delta Lake pipeline to ensure data quality for a customer order dataset. They need to prevent records with negative order amounts from being written into the Delta table. Which approach should they choose, considering the strengths and limitations of various Delta Lake features?
- A
Use Delta Lake's schema enforcement to prevent negative order amounts.
- B
Implement Delta Lake's CHECK constraints to enforce that order amounts are non-negative.
- C
Set up a streaming job to filter out records with negative order amounts before writing to the Delta table.
- D
Leverage Delta Lake's optimistic concurrency control to reject transactions with negative order amounts.
Show answer and explanation
Correct answer: B
Explanation
The best approach for enforcing data quality at the Delta Lake table level is to use CHECK constraints, as they allow you to define rules that data must satisfy before being written to the table. Other options, such as schema enforcement or streaming job filters, do not provide the same level of assurance or are not designed for this purpose.
- A. Incorrect.
Delta Lake's schema enforcement is used to enforce data types and column presence but does not support the enforcement of specific value constraints like preventing negative numbers.
- B. Correct.
Delta Lake's CHECK constraints are designed to enforce data quality rules, such as ensuring that specific column values meet certain conditions (e.g., non-negative order amounts). This is the correct approach for the described scenario.
- C. Incorrect.
While filtering negative values in a streaming job could help, it does not enforce data quality at the Delta Lake table level, making this approach less reliable for ensuring overall data integrity.
- D. Incorrect.
Delta Lake's optimistic concurrency control is used to manage conflicts during concurrent transactions but does not enforce specific data quality rules.