Databricks Machine Learning Associate Question 34
Select 3You are tasked with building a machine learning pipeline in Databricks that includes the following steps: data preprocessing, feature engineering, model training, and model evaluation. These tasks need to be executed in a specific sequence and should be orchestrated efficiently. Which of the following steps are necessary to correctly orchestrate this multi-task workflow using Databricks Jobs?
- A
Define each step as a separate task in a Databricks Job and configure task dependencies.
- B
Execute all tasks manually in the Databricks notebook environment to ensure proper sequencing.
- C
Use the Databricks Workflows UI or API to define and orchestrate the tasks.
- D
Configure retry policies and timeouts for critical tasks within the Databricks Job.
- E
Write custom Spark code for task orchestration instead of using Databricks Jobs.
Show answer and explanation
Correct answers: A, C, D
Explanation
To orchestrate multi-task ML workflows in Databricks, each step in the pipeline should be defined as a separate task in a Databricks Job, and task dependencies should be configured to ensure proper sequencing. The Databricks Workflows UI or API can be used to define these tasks. Additionally, setting retry policies and timeouts for critical tasks enhances the robustness of the workflow. Manual execution or writing custom Spark code for orchestration is not recommended as it is inefficient and does not leverage Databricks' built-in capabilities.
- A. Correct.
Correct. Defining each step as a separate task and configuring task dependencies ensures that tasks are executed in the correct order within the Databricks Job.
- B. Incorrect.
Incorrect. Manually executing tasks in notebooks is not an efficient or scalable way to orchestrate workflows.
- C. Correct.
Correct. The Databricks Workflows UI or API provides a user-friendly and programmatic way to define and orchestrate task workflows.
- D. Correct.
Correct. Configuring retry policies and timeouts for critical tasks ensures robustness and reliability in the workflow execution.
- E. Incorrect.
Incorrect. While custom Spark code can be written for task orchestration, it is unnecessary and less efficient compared to using Databricks Jobs, which are purpose-built for this functionality.