Databricks Data Engineer Associate Question 436
Select 1You are designing a workflow in Databricks Jobs to process data for a customer. The first step involves extracting data from an external API and storing it as a raw file in a Delta table. The second step involves running a transformation job on the raw Delta table to clean the data. Finally, the third step involves running a machine learning model on the cleaned data. In this scenario, which task(s) should be set as a predecessor to the machine learning task?
- A
The data extraction task
- B
The data transformation task
- C
Both the data extraction and transformation tasks
- D
No predecessor task is required
Show answer and explanation
Correct answer: C
Explanation
In a Databricks Jobs workflow, predecessor tasks must be set up to ensure that dependent tasks are executed only after their dependencies are complete. In this scenario, the machine learning task depends on the cleaned data produced by the transformation task, which in turn depends on the data extracted by the extraction task. Therefore, both the extraction and transformation tasks must be set as predecessors to the machine learning task.
- A. Incorrect.
The data extraction task is important, but it alone does not ensure that the data is ready for the machine learning task. Transformations must also be completed first.
- B. Incorrect.
The data transformation task ensures the data is clean, but it depends on the output of the data extraction task. Therefore, both tasks must precede the machine learning task.
- C. Correct.
Both the data extraction and transformation tasks must be completed before the machine learning task can run, as the machine learning task depends on the cleaned data produced after the transformation.
- D. Incorrect.
No predecessor task is required is incorrect because the machine learning task cannot execute without the necessary cleaned data produced by the earlier tasks.