Databricks Data Engineer Associate Question 435
Select 2You are building a data pipeline in Databricks using Delta Live Tables. The pipeline includes the following steps: 1) Ingest raw data into a Bronze table, 2) Clean and transform the data into a Silver table, and 3) Aggregate the data for reporting into a Gold table. At which step should you configure a predecessor task to ensure the pipeline executes in the correct order?
- A
Configure a predecessor task for the Bronze table to depend on the Silver table.
- B
Configure a predecessor task for the Silver table to depend on the Bronze table.
- C
Configure a predecessor task for the Gold table to depend on the Silver table.
- D
No predecessor tasks are required for this pipeline because Delta Live Tables automatically handles dependencies.
Show answer and explanation
Correct answers: B, C
Explanation
In a data pipeline, predecessor tasks should be configured to ensure that dependent steps execute in the correct order. The Silver table must wait for the Bronze table to complete before starting, and the Gold table must wait for the Silver table. While Delta Live Tables can infer dependencies in some cases, explicitly defining them is a best practice for maintaining control over the execution flow, especially in complex pipelines.
- A. Incorrect.
Incorrect: The Bronze table is the first step in the process and does not depend on any upstream task. Configuring the Bronze table to depend on the Silver table would create a circular dependency.
- B. Correct.
Correct: The Silver table depends on the Bronze table because the raw data must be ingested into the Bronze table before it can be cleaned and transformed for the Silver table.
- C. Correct.
Correct: The Gold table depends on the Silver table because the cleaned and transformed data from the Silver table must be available before it can be aggregated into the Gold table.
- D. Incorrect.
Incorrect: While Delta Live Tables can infer some dependencies, explicitly defining predecessor tasks ensures the pipeline executes in the desired order and avoids potential issues in complex workflows.