Databricks Data Engineer Professional Question 74
Select 2You are building a streaming data pipeline using Databricks Structured Streaming and Delta Lake. The pipeline reads real-time clickstream data from a Kafka topic, processes it to compute session-level statistics, and writes the results to a Delta Lake table for downstream consumption. To ensure exactly-once processing and avoid duplicates in the Delta table, which of the following configurations or features should you implement?
- A
Use Delta Lake's
mergeoperation with the appropriate condition to handle duplicate records during upserts. - B
Configure Structured Streaming with a checkpoint directory to maintain streaming state and offsets.
- C
Enable Delta Lake's Optimize command to compact small files and improve performance.
- D
Write the stream in Append mode to the Delta table to ensure low latency.
- E
Use a unique transaction identifier for each write operation to the Delta table.
Show answer and explanation
Correct answers: A, B
Explanation
To ensure exactly-once processing and avoid duplicates in a Delta Lake table, it is essential to use Delta's merge operation to handle duplicate records during upserts and configure Structured Streaming with a checkpoint directory to maintain offsets and state. These two features work together to provide robust data consistency in streaming pipelines. Other options, such as Append mode or Optimize, are useful for performance optimizations but do not directly address the core requirement of avoiding duplicates.
- A. Correct.
Correct: Using Delta Lake's
mergeoperation ensures that duplicate records are handled appropriately during upserts, which is essential for maintaining data integrity in the Delta table. - B. Correct.
Correct: Configuring a checkpoint directory is critical for Structured Streaming to maintain offsets and state, ensuring exactly-once processing semantics.
- C. Incorrect.
Incorrect: While the Optimize command improves performance by compacting small files, it does not directly address the need to avoid duplicates or ensure exactly-once processing in the pipeline.
- D. Incorrect.
Incorrect: Writing in Append mode does not inherently guarantee exactly-once processing. It is suitable for low-latency pipelines but may lead to duplicates if not managed properly.
- E. Incorrect.
Incorrect: Using unique transaction identifiers is not a core feature of Structured Streaming and Delta Lake for avoiding duplicates. Instead, checkpointing and
mergeoperations are more reliable.