Databricks Data Engineer Professional Question 92: You are tasked with implementing deduplication logic in a real-time... | Databricks Data Engineer Professional Dumps

Databricks Certified Data Engineer Professional. Professional level, Databricks. Free question with the correct answer and a full explanation.

Databricks Data Engineer Professional Question 92

Select 2

You are tasked with implementing deduplication logic in a real-time data pipeline using Spark Structured Streaming. The data contains events with the following schema: event_id (unique identifier), event_timestamp (event creation time), and data (payload). You want to deduplicate events by ensuring that only the latest event for each event_id is retained within a sliding 10-minute window. Which of the following Spark Structured Streaming approaches will correctly implement this deduplication logic?

A
Use withWatermark on the event_timestamp column with a 10-minute delay, followed by a dropDuplicates transformation on the event_id column.
B
Use groupBy on the event_id column and aggregate the latest event using the max(event_timestamp) function.
C
Use withWatermark on the event_timestamp column with a 10-minute delay, followed by a groupBy on both event_id and event_timestamp, keeping the latest entry using a window function.
D
Use flatMapGroupsWithState with event deduplication logic implemented in the custom state update function.
E
Use a distinct transformation directly on the input DataFrame to remove duplicate event_id values.

Show answer and explanation

Correct answers: A, D

Explanation

In Spark Structured Streaming, deduplication can be achieved using either built-in transformations like withWatermark and dropDuplicates or by implementing custom stateful logic using flatMapGroupsWithState. The first approach is simpler and leverages the structured streaming engine's optimizations, while the second provides more flexibility and control over the deduplication process. Other options either lack proper handling of late data or are not suitable for streaming scenarios.

A. Correct.
Correct. Using withWatermark ensures that late data beyond the watermark threshold is discarded, and dropDuplicates removes duplicate event_id values while processing the stream.
B. Incorrect.
Incorrect. While groupBy and max(event_timestamp) can identify the latest event, this approach is not suitable for use in streaming pipelines without a watermark to handle late data and a deduplication mechanism.
C. Incorrect.
Incorrect. Grouping by both event_id and event_timestamp would not guarantee deduplication of events, as it may retain multiple rows with the same event_id but different timestamps.
D. Correct.
Correct. flatMapGroupsWithState allows for fine-grained stateful processing where custom logic can be written to track and deduplicate events. This is a valid approach for achieving deduplication in a streaming context.
E. Incorrect.
Incorrect. The distinct transformation is not designed for use in streaming pipelines and does not guarantee proper deduplication over a sliding window.

Databricks Data Engineer Professional practice question 92 of 313

Databricks Data Engineer Professional Question 92

Take a Databricks Data Engineer Professional practice test under exam conditions

More Databricks Data Engineer Professional practice questions