Databricks Data Engineer Associate Question 422
Select 2You are designing a production pipeline in Databricks to process streaming data from a Kafka source. The pipeline must ensure exactly-once processing semantics and should be fault-tolerant in case of node failures. Which of the following configurations and practices should you apply to meet these requirements?
- A
Use a checkpoint directory and enable structured streaming checkpointing.
- B
Configure the Kafka source with the 'latest' starting offset to ensure no data is missed.
- C
Use Delta Lake as the sink for idempotent writes.
- D
Disable automatic retries to minimize reprocessing.
- E
Enable write-ahead logs (WAL) for fault-tolerance.
Show answer and explanation
Correct answers: A, C
Explanation
To design a production pipeline with exactly-once processing semantics in Databricks, checkpointing must be enabled to maintain state and recover from failures, and an idempotent sink like Delta Lake should be used to handle retries without data duplication. These configurations ensure fault-tolerance and consistency across the pipeline.
- A. Correct.
Using a checkpoint directory and enabling checkpointing ensures stateful operations and fault-tolerance in structured streaming. This is an essential practice for exactly-once processing.
- B. Incorrect.
Configuring the Kafka source with the 'latest' starting offset might lead to missed data if the pipeline stops and restarts, which violates exactly-once processing requirements. Instead, 'earliest' or specific offsets should be used depending on the use case.
- C. Correct.
Using Delta Lake as the sink enables idempotent writes, meaning the same data can be written multiple times without duplication, which is critical for ensuring exactly-once semantics.
- D. Incorrect.
Disabling automatic retries would reduce the chances of recovering from temporary failures, which is not recommended for building fault-tolerant pipelines.
- E. Incorrect.
Write-ahead logs (WAL) are not required in Databricks structured streaming pipelines as checkpointing already provides the necessary fault-tolerance mechanisms.