Databricks Data Engineer Professional Question 279
Select 4You are tasked with creating a Databricks Job to process daily incoming data from an external source. The data is ingested into a Delta Lake table, and the Job should perform a series of transformations and write the results into another Delta Lake table for reporting purposes. The requirements are as follows:
- The Job should run automatically every day at 2 AM.
- It should process data incrementally, avoiding duplication of already processed data.
- Ensure that the Job retries up to 3 times in case of transient failures.
Which of the following configurations should you apply when creating the Databricks Job to meet these requirements?
- A
Set the Job schedule to run daily at 2 AM using a cron expression.
- B
Add a Task parameter to configure 'Retry Limit' to 3 retries.
- C
Use a Notebook Task with a Merge operation to ensure idempotent writes into the Delta Lake table.
- D
Enable 'Continuous' mode for the Job to process data in real-time.
- E
Configure a Task dependency to ensure the reporting table is updated only after successful transformation.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
To meet the requirements, the Job must be scheduled to run daily at 2 AM using a cron expression. Incremental processing and idempotency can be achieved using a Merge operation in the Notebook Task. Configuring a Retry Limit of 3 ensures resilience against transient failures, and Task dependencies ensure proper execution order. Real-time processing (e.g., Continuous mode) is not required for this daily batch use case.
- A. Correct.
This is correct because configuring a cron expression to run the Job at 2 AM ensures the Job runs automatically at the required time.
- B. Correct.
This is correct because setting the 'Retry Limit' to 3 ensures the Job retries up to 3 times in case of transient failures as per the requirements.
- C. Correct.
This is correct as using a Merge operation ensures incremental processing and avoids duplication of already processed data, aligning with the requirement.
- D. Incorrect.
This is incorrect because enabling 'Continuous' mode is not necessary for daily batch processing; it is more suited for real-time streaming use cases.
- E. Correct.
This is correct because configuring Task dependencies ensures the reporting table is updated only after successful transformation, meeting the dependency requirement.