Google Professional Data Engineer Question 97
Single answerGoogle Cloud PlatformYou are designing a real-time data processing system using Google Cloud Dataflow to analyze streaming data from IoT sensors. The sensors occasionally send events with significant delays. The analysis requires aggregating data in 5-minute windows and ensuring late-arriving data is included in the aggregation for up to 10 minutes after the window closes. How should you configure the pipeline to handle this requirement?
- A
Set the windowing function to Fixed Windows with a duration of 5 minutes and use a watermark to discard late data.
- B
Set the windowing function to Fixed Windows with a duration of 5 minutes and configure a 10-minute allowed lateness.
- C
Set the windowing function to Sliding Windows with a duration of 5 minutes and use a watermark to discard late data.
- D
Set the windowing function to Global Windows and configure a 10-minute allowed lateness.
Show answer and explanation
Correct answer: B
Explanation
To meet the requirement, Fixed Windows with 5-minute duration must be used to aggregate data within the time interval. Configuring a 10-minute allowed lateness ensures that late-arriving data is still processed and included in the appropriate aggregation window up to 10 minutes after the window closes. This approach balances the need for real-time processing with the handling of delayed events.
- A. Incorrect.
Incorrect: A watermark alone cannot handle late-arriving data as it defines when the system expects all data for a window to arrive, but it does not configure how late data should be handled.
- B. Correct.
Correct: Fixed Windows with 5-minute duration and 10-minute allowed lateness ensures that late-arriving data can be included in the window's aggregation for up to 10 minutes after the window has closed.
- C. Incorrect.
Incorrect: Sliding Windows are unnecessary here because the requirement specifies fixed 5-minute windows, and using a watermark without allowed lateness will discard late data.
- D. Incorrect.
Incorrect: Global Windows aggregate all data into a single window, which does not meet the requirement for 5-minute aggregation windows.