Databricks Data Engineer Professional Question 282
Select 2A Databricks Data Engineer is tasked with creating a Job to process daily sales data. The process involves reading raw data from a Delta table, performing transformations like aggregations and filtering, and saving the results back to another Delta table. The job should be scheduled to run daily at midnight and must retry up to three times in case of failures. Which configuration steps should the Data Engineer take to meet these requirements?
- A
Use the Delta Live Tables feature to define the transformation logic and schedule the pipeline to run daily at midnight.
- B
Create a Databricks Job, specify a Python or SQL notebook for the transformation logic, and configure a daily schedule at midnight with retries set to 3.
- C
Set up a Databricks Workflow with multiple tasks, including a task for reading raw data, a task for transformations, and a task for writing the output, and configure the workflow to run daily at midnight with retries enabled.
- D
Manually run a Databricks notebook containing the transformation logic every day at midnight using a personal login session.
- E
Set the cluster configuration of the Databricks Job to an all-purpose cluster to ensure quick job execution.
Show answer and explanation
Correct answers: B, C
Explanation
To meet the requirements of scheduling a job to run daily at midnight with retries enabled, the Data Engineer can either create a Databricks Job with a notebook as the transformation logic or set up a Databricks Workflow with multiple tasks for modularization. Both approaches allow for the configuration of scheduling and retries, making them suitable solutions for the scenario.
- A. Incorrect.
Delta Live Tables is a powerful feature for declarative ETL pipelines but is not explicitly required in this scenario. The question focuses on creating a Databricks Job with specific scheduling and retry settings, which can be achieved without Delta Live Tables.
- B. Correct.
Creating a Databricks Job with a notebook for transformation logic and configuring a daily schedule with retries is a valid approach and aligns with the requirements of the scenario.
- C. Correct.
Using a Databricks Workflow with multiple tasks for each step of the pipeline is another valid approach that meets the requirements for scheduling and retries.
- D. Incorrect.
Manually running a notebook every day at midnight is not a scalable or automated solution and does not align with best practices.
- E. Incorrect.
Using an all-purpose cluster for job execution is unnecessary and costly. A job cluster is more appropriate for scheduled jobs and ensures resource efficiency.