Databricks Data Engineer Professional Question 96
Select 3You are working on a Spark Structured Streaming application to deduplicate real-time data from a Kafka topic. Each record in the stream contains a unique identifier record_id and a timestamp event_time. To ensure that only unique records are processed within a 10-minute window, which of the following approaches should you implement?
- A
Use a watermark on the
event_timecolumn and apply a dropDuplicates transformation on therecord_idcolumn. - B
Use a sliding window of 10 minutes on the
event_timecolumn and filter out duplicaterecord_idvalues within the window. - C
Use a watermark on the
event_timecolumn and apply a groupBy onrecord_id, selecting the latest record for eachrecord_idbased onevent_time. - D
Apply a dropDuplicates transformation directly on the
record_idcolumn without using a watermark or windowing. - E
Use a stateful aggregation to track processed
record_idvalues and filter out duplicates in subsequent micro-batches.
Show answer and explanation
Correct answers: A, C, E
Explanation
Deduplication in Spark Structured Streaming requires handling late data effectively while ensuring unique records are processed. Using watermarking and transformations like dropDuplicates or groupBy ensures late data is managed, and stateful aggregation allows tracking of duplicates across micro-batches. Options 1, 3, and 5 correctly implement these strategies for deduplication, while options 2 and 4 fail to address key aspects of late data handling or deduplication.
- A. Correct.
Correct: Using a watermark ensures late data is handled effectively, and dropDuplicates can eliminate duplicates based on
record_id. This is a common and efficient approach for deduplication in structured streaming when combined with windowing. - B. Incorrect.
Incorrect: While sliding windows can help group data, they do not inherently deduplicate records, and there is no mention of how duplicates are filtered in this approach.
- C. Correct.
Correct: This approach uses watermarking to handle late data and ensures deduplication by grouping records by
record_idand selecting the latest one based onevent_time. This is a valid deduplication strategy in Spark Structured Streaming. - D. Incorrect.
Incorrect: Applying dropDuplicates without watermarking will not handle late data effectively, leading to possible duplicates being ignored if they arrive later.
- E. Correct.
Correct: Stateful aggregation can track processed
record_idvalues across micro-batches, ensuring that duplicates are filtered out even in the presence of late-arriving data.