Databricks Data Engineer Associate Question 30
Single answerYou are tasked with running a one-time ETL job that processes a large dataset in Databricks. The job is expected to take a few hours to complete, and cost efficiency is a key factor. Which type of cluster should you use for this scenario?
- A
All-purpose cluster
- B
Job cluster
- C
A mix of all-purpose and job clusters
- D
Neither, as clusters are not needed for running ETL jobs in Databricks
Show answer and explanation
Correct answer: B
Explanation
Job clusters in Databricks are best suited for one-time or scheduled tasks like ETL jobs. They are automatically created when the job starts, optimized for cost efficiency, and terminated once the job completes, avoiding unnecessary compute costs. All-purpose clusters, on the other hand, are ideal for interactive and collaborative workloads but are less cost efficient for one-time jobs as they remain active until manually shut down.
- A. Incorrect.
All-purpose clusters are designed for interactive use, such as exploratory data analysis or collaborative development, but they incur higher costs when running one-time jobs as they remain active until manually terminated.
- B. Correct.
Job clusters are specifically designed for running jobs, are automatically terminated after the job finishes, and are more cost efficient for one-time ETL tasks.
- C. Incorrect.
Using a mix of all-purpose and job clusters is not a valid approach for running a single ETL job, as only one cluster type is needed for this task.
- D. Incorrect.
Clusters are required for running ETL jobs in Databricks, as they provide the compute resources necessary for processing data.