Databricks Machine Learning Associate Question 227
Single answerYou are working on tuning a machine learning model in Databricks using Hyperopt. The model is single-node and you want to leverage parallel execution to reduce the total tuning time. Which of the following steps correctly describes how you can achieve this?
- A
Use the SparkTrials class to distribute the execution of trials across a Spark cluster.
- B
Enable Spark parallelism by setting the 'parallelism' parameter in Hyperopt's fmin() function.
- C
Use MLflow to log the tuning trials and enable parallel execution.
- D
Convert the single-node model to a distributed model to utilize Spark parallelism.
Show answer and explanation
Correct answer: A
Explanation
To parallelize the tuning process for single-node models using Hyperopt in Databricks, you must use SparkTrials. This allows the trials to be distributed across the Spark cluster, reducing overall execution time. Other options, such as adjusting the 'parallelism' parameter or using MLflow, do not achieve this parallelization.
- A. Correct.
Correct. SparkTrials is specifically designed to enable parallelization of Hyperopt trials across a Spark cluster, even for single-node models.
- B. Incorrect.
Incorrect. While 'parallelism' is a parameter in Hyperopt's fmin() function, it does not automatically enable Spark parallelism. You need to explicitly use SparkTrials for this.
- C. Incorrect.
Incorrect. MLflow is used for experiment tracking and logging but does not handle parallel execution of trials.
- D. Incorrect.
Incorrect. You do not need to convert a single-node model to a distributed model to leverage Spark parallelism for Hyperopt; SparkTrials handles this for you.