Databricks Data Engineer Associate Question 26
Single answerYou are tasked with running a one-time data pipeline that performs heavy transformations, and you want to optimize cost by using a cluster that automatically terminates after the job completes. Which type of cluster should you use in Databricks?
- A
All-purpose cluster
- B
Jobs cluster
- C
Interactive cluster
- D
Shared cluster
Show answer and explanation
Correct answer: B
Explanation
For one-time data pipelines or workloads, jobs clusters are the ideal choice because they are provisioned specifically for the job and terminate automatically after completion. This minimizes costs compared to all-purpose clusters, which remain active unless manually terminated.
- A. Incorrect.
All-purpose clusters are designed for interactive workloads and shared usage among multiple users. They do not automatically terminate based on job completion, making them less cost-efficient for one-time jobs.
- B. Correct.
Jobs clusters are dedicated for running a specific job or task, and they automatically terminate after the job completes, optimizing costs for one-time workloads.
- C. Incorrect.
Interactive clusters are another term sometimes used for all-purpose clusters, which are ideal for ad-hoc analysis but not cost-efficient for one-time job execution.
- D. Incorrect.
Shared clusters are a configuration of all-purpose clusters that allow multiple users to share the same resources, but they are not designed to terminate after completing a specific job.