Databricks Data Engineer Professional Question 285
Single answerYou are tasked with creating a multi-task job in Databricks to process and transform data. The workflow consists of three tasks: Task A, Task B, and Task C. Task A ingests raw data into a Delta table, Task B performs transformations on the Delta table created by Task A, and Task C generates aggregated reports based on the transformed data. Task B cannot start until Task A completes successfully, and Task C must wait for Task B to finish. How should you configure the job dependencies in Databricks?
- A
Set Task B as dependent on Task A and Task C as dependent on Task B.
- B
Set Task A as dependent on Task B and Task C as dependent on Task A.
- C
Set Task C as dependent on Task A and Task B to run in parallel.
- D
Set all tasks to run independently without specifying dependencies.
Show answer and explanation
Correct answer: A
Explanation
In Databricks, multi-task jobs allow you to define dependencies between tasks to ensure proper execution order. In this scenario, Task B relies on the output of Task A, and Task C relies on the output of Task B. Configuring the dependencies as described in the correct answer ensures the tasks are executed sequentially in the correct order.
- A. Correct.
This is the correct configuration. Task B depends on the completion of Task A, and Task C depends on Task B. This ensures the workflow executes in the required order.
- B. Incorrect.
This is incorrect because Task B should not depend on Task A, and Task C should not depend on Task A directly. This would lead to an incorrect order of execution.
- C. Incorrect.
This is incorrect because Task C cannot run immediately after Task A; it must wait for Task B to complete the transformations.
- D. Incorrect.
This is incorrect because running all tasks independently would violate the required dependencies, leading to potential data inconsistencies or failures.