Databricks Machine Learning Associate Question 35
Select 3You are a data scientist working on a project that involves training multiple machine learning models and performing data pre-processing in sequential and parallel steps. You decide to use Databricks Jobs to orchestrate the workflow. Which of the following steps are necessary to correctly set up a multi-task workflow in Databricks Jobs?
- A
Define individual tasks with their dependencies to represent the workflow structure.
- B
Use a Databricks SQL query as a task to perform data pre-processing before model training.
- C
Manually trigger each task in the workflow to ensure execution in the correct order.
- D
Set up cluster configurations for each task based on its computational requirements.
- E
Enable notifications for job completion to monitor the workflow execution.
Show answer and explanation
Correct answers: A, B, D
Explanation
To orchestrate a multi-task ML workflow in Databricks Jobs, defining tasks with dependencies, using tasks like SQL queries for pre-processing, and configuring clusters for each task are essential. These steps ensure the workflow executes efficiently and as intended. Manually triggering tasks is unnecessary because Databricks handles task execution automatically. Notifications, while helpful, are optional and not central to the workflow setup.
- A. Correct.
Correct: In Databricks Jobs, it is necessary to define tasks and their dependencies to establish the workflow structure for sequential or parallel execution.
- B. Correct.
Correct: Databricks Jobs support running SQL queries as tasks, which can be used for data pre-processing before other tasks like model training.
- C. Incorrect.
Incorrect: Databricks Jobs execute tasks automatically based on the defined dependencies, so manual triggering is not required.
- D. Correct.
Correct: Each task can have its own cluster configuration, allowing you to allocate resources appropriately based on task requirements.
- E. Incorrect.
Incorrect: While enabling notifications is useful for monitoring, it is not a required step to set up a multi-task workflow.