Databricks Data Engineer Professional Question 93
Single answerYou are implementing a Spark Structured Streaming application to process a real-time stream of transaction data. The data contains duplicate records, and your task is to ensure that only unique records are written to the output sink. Each record has a unique transaction_id. How can you implement deduplication in Spark Structured Streaming to achieve this?
- A
Use the
dropDuplicatestransformation on the streaming DataFrame and specify thetransaction_idcolumn. - B
Use the
distincttransformation on the streaming DataFrame. - C
Apply a watermark on the
timestampcolumn and use thedropDuplicatestransformation with thetransaction_idcolumn. - D
Use the
groupByoperation on thetransaction_idcolumn and apply an aggregation to keep only the first occurrence.
Show answer and explanation
Correct answer: C
Explanation
In Spark Structured Streaming, deduplication in real-time streams is best achieved by combining a watermark with the dropDuplicates transformation. The watermark ensures that only data within a specific time window is considered, preventing unbounded state growth. The dropDuplicates transformation removes duplicates based on the specified column (in this case, transaction_id). This approach is optimal for handling streaming data with late arrivals and ensuring memory efficiency.
- A. Incorrect.
The
dropDuplicatestransformation can remove duplicates based on a specific column, but without a watermark, it cannot handle streaming data with late arrivals effectively. It may cause unbounded state growth. - B. Incorrect.
The
distincttransformation removes duplicates across the entire DataFrame but does not handle state management or late-arriving data in a streaming context. - C. Correct.
Using a watermark on the
timestampcolumn ensures that old data outside the watermark window is discarded from the state. Combined withdropDuplicateson thetransaction_idcolumn, this approach effectively deduplicates records while managing memory efficiently in a streaming context. - D. Incorrect.
While
groupByand aggregation can be used to deduplicate data, it is not the most efficient or recommended method for this use case in Spark Structured Streaming. It requires additional processing and state management logic.