Databricks Data Engineer Associate Question 428
Single answerYou are tasked with building a Databricks Job that processes data in three stages: data ingestion, data transformation, and reporting. Each stage is implemented as a separate task within the job. You want to ensure that the 'data transformation' task starts only after the 'data ingestion' task is successfully completed. How can you configure this dependency using Databricks Jobs?
- A
Set the 'data ingestion' task as a predecessor of the 'data transformation' task in the job configuration.
- B
Use a Databricks REST API call to link the 'data ingestion' and 'data transformation' tasks programmatically.
- C
Configure both the 'data ingestion' and 'data transformation' tasks to run simultaneously by default.
- D
Manually trigger the 'data transformation' task after verifying the completion of the 'data ingestion' task.
Show answer and explanation
Correct answer: A
Explanation
To configure task dependencies in Databricks Jobs, you can specify predecessor tasks directly within the job configuration. This ensures that tasks are executed in the correct sequence, such as 'data transformation' starting only after 'data ingestion' is completed. This approach simplifies workflow management and avoids manual intervention or custom scripting.
- A. Correct.
This is correct. In Databricks Jobs, you can set task dependencies using the job configuration interface, where you specify predecessor tasks to ensure tasks run in the correct order.
- B. Incorrect.
This is incorrect. While you can use the REST API to manage jobs programmatically, the question specifically asks about setting up a dependency within the job configuration itself.
- C. Incorrect.
This is incorrect. Configuring tasks to run simultaneously would not enforce the dependency between 'data ingestion' and 'data transformation' tasks.
- D. Incorrect.
This is incorrect. Manually triggering tasks does not leverage the automated dependency management provided by Databricks Jobs.