Databricks Machine Learning Professional Question 101
Select 3You are working on a machine learning workflow in Databricks where you need to run a batch inference job on a large dataset. You are considering whether to use a Job cluster or an all-purpose cluster. What are the advantages of using a Job cluster for this task?
- A
Job clusters automatically terminate after the job is complete, reducing costs.
- B
Job clusters provide better runtime performance compared to all-purpose clusters for all workloads.
- C
Job clusters are dedicated to specific tasks, ensuring no interference from other workloads.
- D
Job clusters allow for reusable cluster configurations for multiple interactive users.
- E
Job clusters can be customized for each job, enabling optimized resource utilization.
Show answer and explanation
Correct answers: A, C, E
Explanation
Job clusters in Databricks are designed for single-use, ephemeral workloads like batch processing or scheduled tasks. They offer benefits such as automatic termination, task isolation, and customizable configurations, making them cost-effective and resource-efficient for job-specific tasks. Unlike all-purpose clusters, they are not intended for interactive or multi-user workflows.
- A. Correct.
Correct: Job clusters automatically terminate once the job finishes, which helps reduce unnecessary costs by not keeping the cluster running when it is no longer needed.
- B. Incorrect.
Incorrect: While Job clusters are optimized for job execution, they do not inherently guarantee better performance for all workloads compared to all-purpose clusters. Performance depends on the workload and cluster configuration.
- C. Correct.
Correct: Job clusters are isolated and dedicated to the specific job they are created for, avoiding interference from other workloads, unlike all-purpose clusters which may host multiple concurrent tasks.
- D. Incorrect.
Incorrect: Job clusters are typically ephemeral and designed for single-use, meaning they are not intended for reuse across multiple users or sessions. All-purpose clusters are better suited for this purpose.
- E. Correct.
Correct: Job clusters can be configured specifically for the requirements of a given job, allowing you to allocate resources optimally and avoid over- or under-provisioning.