Databricks Data Engineer Associate Question 28
Single answerA data engineering team is building a data pipeline in Databricks to process large volumes of data every day. The pipeline runs automatically on a schedule and requires a cost-effective approach. Which type of cluster should the team use, and why?
- A
All-purpose cluster, because it supports multiple workloads and interactive development.
- B
Jobs cluster, because it is created specifically for the scheduled run and terminates automatically after execution.
- C
All-purpose cluster, because it ensures high availability for scheduled jobs.
- D
Jobs cluster, because it can run multiple workloads simultaneously and supports interactive development.
Show answer and explanation
Correct answer: B
Explanation
Jobs clusters are tailored for non-interactive, scheduled workloads like data pipelines. They are provisioned on-demand and terminate when the job completes, minimizing costs. In contrast, all-purpose clusters are suited for interactive workloads but are less cost-efficient for scheduled jobs since they remain active indefinitely unless manually terminated.
- A. Incorrect.
All-purpose clusters are intended for interactive development, such as running notebooks or ad-hoc queries. They are not the most cost-effective option for scheduled pipelines due to their long-running nature.
- B. Correct.
Jobs clusters are designed for scheduled job execution. They are created on-demand, terminate automatically after the job completes, and are more cost-effective for such use cases.
- C. Incorrect.
While all-purpose clusters can technically run scheduled jobs, they are not optimized for cost and resource usage in this scenario as they remain active even when not in use.
- D. Incorrect.
Jobs clusters are not designed to support multiple workloads simultaneously or interactive development. They are created for specific job executions, making this option incorrect.