Databricks Data Engineer Professional Question 95
Select 2You are tasked with implementing a deduplication strategy for a Spark Structured Streaming job that processes a stream of events. Each event is uniquely identified by a combination of the 'event_id' and 'timestamp' fields. To ensure only the newest records for each 'event_id' are retained in the output, which approach should you use?
- A
Use a watermark on the 'timestamp' column and drop duplicates based on the 'event_id' column.
- B
Use the
dropDuplicatesmethod on both 'event_id' and 'timestamp' columns. - C
Maintain the latest record for each 'event_id' using a stateful processing mechanism.
- D
Use the
distinctmethod to remove duplicates based on all columns in the dataset.
Show answer and explanation
Correct answers: A, C
Explanation
To deduplicate a Spark Structured Streaming job based on unique identifiers like 'event_id' while retaining the latest record, you can use a combination of watermarks and stateful processing. Watermarks enable the system to manage state efficiently by discarding old data, while stateful processing or the dropDuplicates method (with appropriate columns) ensures deduplication is performed correctly. The distinct method is not suitable for this case as it does not address event recency or stateful deduplication.
- A. Correct.
This is correct. Using a watermark on the 'timestamp' column ensures the system can clean up state for older events, while dropping duplicates on the 'event_id' column ensures only the most recent records for each event are retained.
- B. Incorrect.
This is incorrect. The
dropDuplicatesmethod works well for static datasets, but in structured streaming, it does not provide a mechanism to handle stateful deduplication with time constraints. - C. Correct.
This is correct. Stateful processing can track the latest record for each 'event_id' across micro-batches, ensuring deduplication is performed accurately in a streaming context.
- D. Incorrect.
This is incorrect. The
distinctmethod removes duplicates based on all columns but does not account for retaining the latest record based on the 'timestamp' field in a streaming scenario.