Databricks Data Engineer Associate Question 437
Select 2You are designing a data pipeline in Databricks using Jobs. The pipeline involves three tasks: Task A ingests raw data from a source, Task B processes the data, and Task C writes the final processed data to a data warehouse. Given the dependencies between these tasks, when should a predecessor task be set up?
- A
Set Task B as a predecessor to Task A to ensure data ingestion happens after processing.
- B
Set Task A as a predecessor to Task B to ensure data is ingested before processing begins.
- C
Set Task C as a predecessor to Task A to ensure the data is written to the data warehouse before ingestion starts.
- D
Set Task B as a predecessor to Task C to ensure the data is processed before being written to the data warehouse.
Show answer and explanation
Correct answers: B, D
Explanation
In a data pipeline, predecessor tasks are set up to enforce dependencies between tasks to ensure they execute in the correct order. Task A (data ingestion) must complete before Task B (data processing) can begin, and Task B must complete before Task C (writing to the data warehouse) can start. This ensures that each task has the necessary input data before execution.
- A. Incorrect.
This is incorrect. Task B cannot be a predecessor to Task A, as data processing depends on data ingestion being completed first.
- B. Correct.
This is correct. Task A must finish before Task B can begin, as Task B depends on the raw data ingested by Task A.
- C. Incorrect.
This is incorrect. Task C cannot be a predecessor to Task A, as writing to the data warehouse depends on the processed data being available, which happens after ingestion and processing.
- D. Correct.
This is correct. Task B must finish before Task C can begin, as Task C depends on the processed data output by Task B.