Databricks Data Engineer Professional Question 4
Select 3You are working on a Databricks workspace and need to schedule a job to process data daily. The job should run a notebook that performs ETL operations. During this process, you want to ensure that the job leverages a cluster that terminates automatically after the job completes to save costs. Which combination of steps should you follow to achieve this?
- A
Create a new Databricks Job and attach the ETL notebook to the job.
- B
Configure a new job cluster for the Databricks Job and set an auto-termination timeout.
- C
Attach the ETL notebook to an existing interactive cluster and manually schedule the job using a cron expression.
- D
Use Databricks CLI to create and schedule the job, ensuring the cluster configuration includes auto-termination.
- E
Enable the 'Retry on Failure' option for the job to ensure reliability in case of errors.
Show answer and explanation
Correct answers: A, B, D
Explanation
To schedule an automated ETL job in Databricks and ensure cost efficiency, you need to create a Databricks Job, attach the ETL notebook, and configure a new job cluster with auto-termination enabled. This ensures that the cluster terminates after the job completes, saving costs. Additionally, using the Databricks CLI allows for programmatic job creation and scheduling while maintaining the required cluster configuration.
- A. Correct.
This is correct because creating a Databricks Job and attaching the ETL notebook is the first step in automating the ETL workflow.
- B. Correct.
This is correct because configuring a new job cluster with an auto-termination timeout ensures cost efficiency by shutting down the cluster after the job completes.
- C. Incorrect.
This is incorrect because attaching a notebook to an existing interactive cluster and manually scheduling the job is not a best practice for cost efficiency or automation.
- D. Correct.
This is correct because the Databricks CLI can be used to create and schedule jobs programmatically while including job cluster configurations such as auto-termination.
- E. Incorrect.
This is incorrect because enabling the 'Retry on Failure' option improves reliability but does not address the core requirement of cost efficiency through cluster auto-termination.