Databricks Data Engineer Professional Question 284
Select 2You are tasked with creating a multi-task job in Databricks to process and analyze customer data. The workflow consists of the following tasks:
- Task A: Load raw data from an external source into a Delta table.
- Task B: Perform data cleaning using a notebook, which depends on Task A.
- Task C: Generate summary metrics using a Python script, which depends on Task B.
- Task D: Send an email notification upon successful completion of Task C.
Which of the following steps will correctly configure the multi-task job and ensure task dependencies are met?
- A
Set Task B to depend on Task A, Task C to depend on Task B, and Task D to depend on Task C.
- B
Configure Task A and Task C to run in parallel, with Task B dependent on Task A and Task D dependent on Task C.
- C
Create a linear dependency by setting Task B to depend on Task A, Task C to depend on Task B, and Task D to depend on Task C.
- D
Set all tasks to run independently, as dependencies are managed automatically by Databricks.
- E
Use the 'Run Now' option for Task D to manually trigger it after Task C completes.
Show answer and explanation
Correct answers: A, C
Explanation
In Databricks multi-task jobs, tasks need to be explicitly configured with dependencies to ensure proper execution order. For the given scenario, Task A must complete before Task B starts, Task B must complete before Task C starts, and Task C must complete before Task D starts. Setting up the dependencies as described in options 1 and 3 ensures the workflow is executed correctly.
- A. Correct.
This is correct as it ensures proper dependency chaining (Task B -> Task A, Task C -> Task B, Task D -> Task C) for the multi-task job.
- B. Incorrect.
This is incorrect because Task C cannot run in parallel with Task A, as it depends on the output of Task B.
- C. Correct.
This is correct because it describes the correct linear dependency configuration for the tasks in a multi-task job.
- D. Incorrect.
This is incorrect because setting all tasks to run independently will break the required dependencies.
- E. Incorrect.
This is incorrect because manually triggering Task D using 'Run Now' is not a valid way to configure dependencies in a multi-task job.