Databricks Machine Learning Professional Question 102
Single answerYou are designing a machine learning workflow on Databricks and need a cluster to execute a batch inference job that runs once daily. Cost optimization and automated resource management are key considerations for your team. Why might you choose a Job cluster over an All-purpose cluster for this task?
- A
Job clusters automatically shut down after the job is completed, reducing costs.
- B
Job clusters are pre-configured with libraries required for most machine learning tasks.
- C
Job clusters can be reused across multiple jobs, improving resource utilization.
- D
Job clusters provide better scalability compared to All-purpose clusters.
Show answer and explanation
Correct answer: A
Explanation
Job clusters are specifically designed for single-job execution and are automatically terminated once the job is finished, making them an excellent choice for cost optimization in tasks like batch inference jobs. In contrast, All-purpose clusters are designed for long-running, interactive workloads, which can lead to higher costs if not actively managed.
- A. Correct.
Job clusters are ephemeral and are automatically terminated after the job execution is complete, which helps minimize costs by avoiding unnecessary resource usage.
- B. Incorrect.
This is incorrect because libraries must still be explicitly configured or installed for Job clusters, just like All-purpose clusters.
- C. Incorrect.
This is incorrect as Job clusters are designed for single-use and are not intended to be reused across multiple jobs. All-purpose clusters are better suited for shared, interactive workloads.
- D. Incorrect.
This is incorrect because both Job clusters and All-purpose clusters can scale similarly, depending on the job requirements and configurations.