Google Professional Data Engineer Question 96
Single answerGoogle Cloud PlatformYou are designing a data pipeline on Google Cloud that processes real-time events from IoT devices. The pipeline should handle late-arriving data and calculate the average temperature readings for each device in 5-minute tumbling windows. Which approach should you take to ensure accurate results?
- A
Use Apache Beam with fixed-time windows and a default watermark, and discard events arriving late.
- B
Use Apache Beam with fixed-time windows, set a late data allowed duration, and apply a trigger that fires at the end of the window.
- C
Use Apache Beam with session windows and allow late data indefinitely.
- D
Use Apache Beam with global windows and triggers to process late-arriving data.
Show answer and explanation
Correct answer: B
Explanation
Tumbling (fixed-time) windows are ideal for calculating averages over fixed intervals like 5 minutes. Allowing a late data duration ensures that late-arriving events are included in the computation, maintaining data accuracy. The trigger ensures that results are emitted at the end of each window, fitting the requirement for timely and accurate calculations.
- A. Incorrect.
This option discards late-arriving events, which is not suitable for handling late data. It would lead to inaccurate results if late events are expected.
- B. Correct.
This option is correct because fixed-time windows (tumbling windows) break the stream into discrete intervals, and setting a late data allowed duration ensures late events are included for accurate calculations. The trigger ensures results are emitted at the end of each window.
- C. Incorrect.
Session windows are not appropriate for this scenario as they group events based on periods of activity, not fixed intervals. Additionally, allowing late data indefinitely can lead to memory issues.
- D. Incorrect.
Global windows are not suitable here as they do not partition data into fixed intervals, which is required for calculating averages in 5-minute windows.