Databricks Data Engineer Professional Question 286
Select 2You are tasked with creating a multi-task job in Databricks to process and analyze customer transaction data. The workflow consists of three tasks: 1) ingesting data from a cloud storage location, 2) transforming the ingested data using a Python notebook, and 3) running an SQL query to generate reports. The transformation task should only start after the ingestion task successfully completes, and the reporting task should depend on the successful completion of the transformation task. Which of the following steps are necessary to correctly configure the dependencies in the job?
- A
Set the transformation task to depend on the ingestion task in the job configuration.
- B
Configure the reporting task to depend on both the ingestion and transformation tasks.
- C
Enable retries for each task in case of failure to ensure job robustness.
- D
Define the reporting task to depend solely on the transformation task in the job configuration.
- E
Schedule all tasks to start concurrently and handle dependencies in the code logic.
Show answer and explanation
Correct answers: A, D
Explanation
To correctly configure a multi-task job with dependencies in Databricks, each task must explicitly define its dependencies in the job configuration. The transformation task should depend on the ingestion task, and the reporting task should depend only on the transformation task. This ensures a proper dependency chain where tasks execute sequentially as required.
- A. Correct.
Correct. The transformation task must explicitly depend on the ingestion task to ensure it starts only after ingestion completes successfully.
- B. Incorrect.
Incorrect. The reporting task should depend only on the transformation task, not both ingestion and transformation, as ingestion is already indirectly covered through the dependency chain.
- C. Incorrect.
Incorrect. While enabling retries is a good practice for robustness, it does not address the need to correctly configure task dependencies in the job.
- D. Correct.
Correct. The reporting task should depend solely on the transformation task, as the transformation task already depends on the ingestion task, ensuring proper execution order.
- E. Incorrect.
Incorrect. Scheduling tasks to start concurrently would bypass the dependency requirements and may lead to execution failures or incorrect results.