Databricks Data Engineer Associate Question 29
Single answerA data engineering team is working on a nightly ETL pipeline that processes large amounts of data. They want to optimize costs and ensure the pipeline runs efficiently without manual intervention. Which type of Databricks cluster should they use in this scenario?
- A
All-purpose cluster
- B
Jobs cluster
- C
Interactive cluster
- D
High-concurrency cluster
Show answer and explanation
Correct answer: B
Explanation
Jobs clusters are best suited for running scheduled tasks like ETL pipelines because they are provisioned specifically for the job and terminate automatically after the job is completed. This makes them cost-efficient and ensures resources are only used when necessary, which aligns with the team's goal of running a nightly ETL pipeline efficiently.
- A. Incorrect.
All-purpose clusters are designed for interactive and collaborative tasks, such as exploration, development, and ad-hoc analysis. They are not optimized for cost-efficiency in scheduled jobs.
- B. Correct.
Jobs clusters are specifically designed to run automated, scheduled tasks like ETL pipelines. They terminate automatically after the job completes, making them more cost-efficient for this scenario.
- C. Incorrect.
Interactive clusters are another name for all-purpose clusters, which makes them suitable for development and exploration, but not optimized for scheduled ETL jobs.
- D. Incorrect.
High-concurrency clusters are useful for serving multiple concurrent users and jobs, but they are not ideal for a single scheduled ETL pipeline as they are more expensive and optimized for multi-user scenarios.