Databricks Data Engineer Associate Question 419
Select 3A data engineering team is tasked with setting up a production pipeline in Databricks to process streaming data from a source system. The team wants the pipeline to be reliable, automatically recover from failures, and ensure data consistency. Which of the following steps are essential to achieve these goals?
- A
Enable checkpointing for the streaming queries.
- B
Use Delta Lake for the sink to ensure ACID compliance.
- C
Schedule the pipeline using a Databricks Job with an appropriate retry policy.
- D
Disable automatic retries to avoid duplicate processing.
- E
Store intermediate data in non-durable storage for faster performance.
Show answer and explanation
Correct answers: A, B, C
Explanation
To build reliable production pipelines in Databricks, enabling checkpointing ensures the pipeline can recover its state after failures. Using Delta Lake guarantees ACID compliance, providing data consistency. Scheduling the pipeline as a Databricks Job with retry policies helps handle transient errors effectively. Disabling retries and using non-durable storage would compromise the reliability and durability of the pipeline.
- A. Correct.
Checkpointing is essential in streaming pipelines to allow stateful operations and to recover from failures without data loss.
- B. Correct.
Delta Lake provides ACID compliance, ensuring data consistency and reliability in the production pipeline.
- C. Correct.
Scheduling the pipeline using a Databricks Job with retry policies ensures the pipeline can recover from transient failures automatically.
- D. Incorrect.
Disabling automatic retries can lead to pipeline failures during transient errors, making it unreliable in production environments.
- E. Incorrect.
Using non-durable storage for intermediate data risks data loss during failures, making it unsuitable for production-grade pipelines.