Databricks Data Engineer Associate Question 27
Single answerYou are tasked with creating a Databricks cluster for a one-time ETL job that processes massive amounts of data and needs to automatically terminate once the job completes. Which cluster type should you choose, and why?
- A
All-purpose cluster, because it can handle both interactive and scheduled workloads.
- B
All-purpose cluster, because it is automatically terminated when the job completes.
- C
Jobs cluster, because it is optimized for one-time tasks and can be configured to terminate after completing the job.
- D
Jobs cluster, because it is designed for long-running, multi-user collaborative workloads.
Show answer and explanation
Correct answer: C
Explanation
Jobs clusters and all-purpose clusters are designed for different purposes in Databricks. Jobs clusters are optimized for transient, one-time tasks such as batch processing or ETL jobs. They can be configured to terminate automatically after job completion, which helps save costs. In contrast, all-purpose clusters are intended for interactive, multi-user workloads and do not automatically terminate when a job is completed. For a one-time ETL job that requires automatic termination, a jobs cluster is the correct choice.
- A. Incorrect.
All-purpose clusters are suitable for interactive and collaborative workloads but are not optimized for one-time tasks or automatic termination.
- B. Incorrect.
All-purpose clusters do not have automatic termination when a job completes, which makes them unsuitable for this scenario.
- C. Correct.
Jobs clusters are specifically designed for one-time tasks like ETL jobs and can be configured to terminate automatically, making them the ideal choice for this use case.
- D. Incorrect.
Jobs clusters are not designed for long-running, collaborative workloads; that is a use case better suited for all-purpose clusters.