Databricks Data Engineer Professional Question 281
Select 3A data engineering team is tasked with setting up an automated ETL pipeline in Databricks to ingest data from a cloud storage bucket, transform it using a series of notebooks, and store the results in a Delta table. They want to schedule this pipeline to run daily and ensure that failures are logged for debugging purposes. Which features should the team use when configuring the Databricks Job to meet these requirements?
- A
Set the task retries and timeout to handle failures and prevent long-running tasks.
- B
Enable the 'Run Now' trigger to schedule the job on a daily basis.
- C
Create a job cluster with autoscaling enabled to optimize resource usage during job runs.
- D
Add email notifications for job success or failure to keep the team informed.
- E
Ensure that all transformation notebooks are executed using the 'All-purpose cluster' to simplify execution.
Show answer and explanation
Correct answers: A, C, D
Explanation
To create a robust and automated ETL pipeline in Databricks, the team should configure retries and timeouts to handle transient failures, use a cost-efficient job cluster with autoscaling, and enable email notifications to stay informed about job statuses. Scheduling should be done using the built-in scheduling feature, not 'Run Now,' and production jobs should avoid using all-purpose clusters to optimize costs and ensure isolation.
- A. Correct.
Correct: Configuring task retries and timeouts ensures that transient issues are handled gracefully and prevents tasks from running indefinitely.
- B. Incorrect.
Incorrect: The 'Run Now' trigger is for ad-hoc execution of jobs, not for scheduling. The team should use a cron expression or the scheduling feature for daily runs.
- C. Correct.
Correct: Using job clusters with autoscaling ensures that resources are allocated efficiently and cost-effectively during the job run.
- D. Correct.
Correct: Adding email notifications alerts the team about job status, making it easier to quickly address failures or confirm successful runs.
- E. Incorrect.
Incorrect: All-purpose clusters are not recommended for production jobs due to higher costs and lack of isolation. Job clusters are better suited for this use case.