Databricks Data Engineer Professional Question 171
Select 4You are designing a data pipeline in Databricks that ingests customer and order data into Delta tables. Since Delta Lake does not enforce foreign key constraints, how can you ensure the integrity of the data when joining these tables?
- A
Use a batch job to validate the relationships between the tables by checking for orphaned records.
- B
Implement a Delta Live Tables pipeline with constraints defined in the SQL queries to enforce data integrity during ingestion.
- C
Rely on Delta Lake's built-in foreign key constraint enforcement to validate the data.
- D
Use a merge operation to remove orphaned records from the tables during periodic updates.
- E
Create unit tests for the data pipeline to validate data relationships during development.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
Since Delta Lake does not enforce foreign key constraints, data engineers must implement alternative strategies to maintain data integrity. These include batch jobs to validate relationships, using Delta Live Tables for constraint-like logic, cleaning data with merge operations, and creating unit tests to catch issues during development. Relying on non-existent built-in foreign key constraint enforcement is not a valid approach.
- A. Correct.
This is correct. A batch job can be designed to periodically validate the relationships, such as ensuring there are no orphaned order records without a corresponding customer.
- B. Correct.
This is correct. Delta Live Tables allows you to define transformation logic that can include constraints or filters to ensure that only valid data is ingested.
- C. Incorrect.
This is incorrect. Delta Lake currently does not have built-in support for enforcing foreign key constraints, so relying on this is not a valid solution.
- D. Correct.
This is correct. Using a merge operation, you can clean and update the tables, removing records that do not satisfy the foreign key relationship.
- E. Correct.
This is correct. Creating unit tests to check the relationships during the development phase ensures that data integrity issues are caught early.