Databricks Data Engineer Associate Question 434
Select 1You are designing a data pipeline in Databricks using Jobs. The pipeline consists of three tasks: Task A performs raw data ingestion from a source system, Task B cleanses and transforms the ingested data, and Task C writes the transformed data to a production database. Which of the following best describes the correct configuration of task dependencies?
- A
Set Task A as the predecessor of Task B and Task B as the predecessor of Task C.
- B
Set Task A as the predecessor of both Task B and Task C.
- C
Set Task B as the predecessor of Task A and Task C as the predecessor of Task B.
- D
Set Task A as the predecessor of Task B, and Task C should not have any predecessor.
- E
Do not configure any task dependencies as Databricks Jobs automatically handle task execution order.
Show answer and explanation
Correct answer: A
Explanation
In this scenario, Task A ingests raw data, Task B transforms the data, and Task C writes the transformed data to a production database. Task dependencies must be configured to ensure the correct execution order: Task A must complete before Task B starts, and Task B must complete before Task C starts. This ensures data flows correctly through the pipeline without any issues.
- A. Correct.
Correct: Task A must complete before Task B can start because Task B depends on the data ingested by Task A. Similarly, Task B must complete before Task C can start because Task C depends on the transformed data produced by Task B.
- B. Incorrect.
Incorrect: Task C cannot start immediately after Task A because it depends on the output of Task B, which transforms the data.
- C. Incorrect.
Incorrect: Task B cannot precede Task A since Task B relies on the data ingested by Task A. Additionally, Task C cannot precede Task B for the same reason.
- D. Incorrect.
Incorrect: Task C depends on the transformed data from Task B, so it should have Task B as a predecessor.
- E. Incorrect.
Incorrect: Databricks Jobs does not automatically infer task execution order; dependencies must be explicitly defined.