Databricks Data Engineer Professional Question 287
Select 1You are tasked with creating a Databricks multi-task job to process raw data, transform it, and load it into a data warehouse. The workflow has the following steps:
- Task A: Ingest raw data into Delta Lake.
- Task B: Perform data transformations on the ingested data.
- Task C: Validate the transformed data.
- Task D: Load the validated data into a data warehouse.
Task B depends on Task A, Task C depends on Task B, and Task D depends on Task C. How should you configure the multi-task job dependencies?
- A
Set Task B to depend on Task A, Task C to depend on Task B, and Task D to depend on Task C.
- B
Configure Task A, Task B, Task C, and Task D to run in parallel without dependencies.
- C
Set Task D to depend on Task A, and configure Task B and Task C to run independently.
- D
Set Task B to depend on Task A, Task C to depend on Task B, and Task D to depend on both Task B and Task C.
Show answer and explanation
Correct answer: A
Explanation
To correctly configure a multi-task job with dependencies in Databricks, each task must be set up to follow the correct sequence of operations. Task B depends on Task A, Task C depends on Task B, and Task D depends on Task C. This sequential dependency ensures that data is ingested, transformed, validated, and loaded in the proper order without skipping any critical steps.
- A. Correct.
This is the correct setup since Task B depends on the output of Task A, Task C depends on Task B, and Task D depends on Task C. This ensures the workflow runs in the correct sequence.
- B. Incorrect.
This is incorrect because running all tasks in parallel ignores the dependencies between the steps. For example, Task B cannot proceed without Task A being completed.
- C. Incorrect.
This is incorrect because Task D depends on the validated data from Task C, so Task C cannot be skipped. Additionally, Task B and Task C must follow Task A in sequence.
- D. Incorrect.
This is incorrect because Task D should only depend on Task C, not Task B directly. Task C ensures data validation before the load step, and skipping this would violate the workflow requirements.