Databricks Data Engineer Professional Question 288
Select 2You have been tasked with creating a multi-task job in Databricks to process a dataset in three sequential steps: data ingestion, data transformation, and data validation. Each task must only begin after the previous one is successfully completed. Additionally, the data validation task should notify an external system upon completion. Which of the following configurations would ensure the job meets these requirements?
- A
Set up the multi-task job with three tasks, each having a dependency on the previous task, and configure the data validation task to include a webhook notification.
- B
Configure the tasks to run concurrently without dependencies and add a final task to notify the external system.
- C
Define the tasks in the job with linear dependencies, ensuring the data validation task executes last, and add a task notification to the data validation task.
- D
Define the tasks with dependencies such that all tasks are executed simultaneously, and configure the data validation task with an external system call.
- E
Set up a multi-task job with the first two tasks running in parallel and configure the data validation task to only depend on the completion of the transformation step.
Show answer and explanation
Correct answers: A, C
Explanation
To create a multi-task job with sequential dependencies, tasks must be configured to run one after another by defining dependencies between them. The data validation task must execute last and handle notifications to the external system. Options 1 and 3 correctly implement these requirements by ensuring dependencies and notifications are properly configured.
- A. Correct.
Correct: Setting up dependencies ensures that each task starts only after the previous one completes. Configuring a webhook notification in the data validation task aligns with the requirement to notify an external system.
- B. Incorrect.
Incorrect: Running tasks concurrently without dependencies does not ensure sequential execution. Adding a final task for notification does not meet the requirement for the data validation task to handle notifications.
- C. Correct.
Correct: Linear dependencies guarantee sequential execution of tasks. Adding a task notification to the data validation task directly satisfies the requirement for external system notification.
- D. Incorrect.
Incorrect: Configuring tasks to execute simultaneously does not meet the requirement for sequential execution. External system calls must be tied to the data validation task, not arbitrary simultaneous execution.
- E. Incorrect.
Incorrect: Running tasks in parallel violates the requirement for sequential processing. The data validation task should depend on both preceding tasks, not just the transformation step.