Databricks Machine Learning Associate Question 31
Single answerYou are tasked with building a machine learning pipeline using Databricks Jobs. The pipeline includes three tasks: data preprocessing (Task A), model training (Task B), and model evaluation (Task C). Task B should only start after Task A completes, while Task C should only start after Task B completes. How should you configure the Databricks Job to orchestrate this workflow?
- A
Create a single task in the Databricks Job that combines all three steps in one notebook.
- B
Set up three separate tasks in the Databricks Job, with Task B depending on Task A and Task C depending on Task B.
- C
Use Databricks Workflows to manually trigger each task in the correct order after the previous task completes.
- D
Configure Task A, Task B, and Task C to run in parallel in the Databricks Job.
Show answer and explanation
Correct answer: B
Explanation
Databricks Jobs allow you to orchestrate multi-task ML workflows by defining tasks and their dependencies. In this scenario, setting up separate tasks for data preprocessing, model training, and model evaluation, and defining dependencies between these tasks ensures that the workflow executes in the correct order. This approach is modular, scalable, and takes full advantage of Databricks Jobs' orchestration capabilities.
- A. Incorrect.
Combining all three steps in one notebook would reduce modularity and make debugging or scaling specific steps of the pipeline more difficult. It is not an optimal approach for orchestrating workflows in Databricks Jobs.
- B. Correct.
This is the correct answer. Configuring separate tasks with dependencies ensures that the tasks are executed in the correct order, and the workflow is modular, maintainable, and scalable.
- C. Incorrect.
Manually triggering tasks defeats the purpose of using Databricks Jobs, which automate task orchestration and execution.
- D. Incorrect.
Running all tasks in parallel would not respect the dependencies between tasks, leading to incorrect execution of the pipeline.