Databricks Data Engineer Professional Question 94
Select 3You are working on a Spark Structured Streaming pipeline that processes real-time clickstream data from a Kafka source. Each event contains the columns event_id, user_id, timestamp, and event_type. Due to potential duplicates in the incoming data, you need to ensure that only the most recent event per event_id is retained in the output. Which of the following approaches can you use to implement deduplication in this scenario?
- A
Use the
withWatermarkmethod on the streaming DataFrame and then apply adropDuplicatestransformation specifying theevent_idcolumn. - B
Use the
groupBytransformation on theevent_idcolumn and apply an aggregation function likemaxon thetimestampcolumn to retain the latest data. - C
Apply a
distincttransformation on the streaming DataFrame to remove duplicate rows. - D
Use the
mapGroupsWithStatefunction to maintain state for eachevent_idand filter out older events based on thetimestampvalue.
Show answer and explanation
Correct answers: A, B, D
Explanation
Deduplication in Spark Structured Streaming can be achieved using several methods. The withWatermark and dropDuplicates combination works for event-time-based deduplication within a defined watermark period. Alternatively, aggregation functions like max on grouped data can achieve deduplication by retaining the most recent record. For more complex scenarios, mapGroupsWithState provides flexibility by maintaining custom state logic to filter older events. The distinct transformation, however, does not meet the requirements for deduplication based on specific keys or retaining the most recent event.
- A. Correct.
Correct. Using
withWatermarkensures late data is managed within a defined event time window, anddropDuplicatescan be applied to deduplicate based on specific columns, such asevent_id. - B. Correct.
Correct. Grouping by
event_idand using an aggregation function likemaxon thetimestampcolumn ensures that only the most recent event for eachevent_idis retained. - C. Incorrect.
Incorrect. The
distincttransformation only removes exact duplicates across all columns in the DataFrame but does not handle deduplication based on specific keys likeevent_idor retain the most recent event. - D. Correct.
Correct. The
mapGroupsWithStatefunction allows you to maintain stateful information for eachevent_idand explicitly filter older events based on thetimestampvalue, which is a valid deduplication strategy in Structured Streaming.