Google Professional Data Engineer Question 98
Single answerGoogle Cloud PlatformYou are designing a real-time data pipeline on Google Cloud to process clickstream data from a website. The data needs to be analyzed using fixed-time windows of 1 minute for aggregations such as page views per minute. However, some events are expected to arrive late by up to 5 minutes due to intermittent network delays. How should you handle late-arriving data in your pipeline to ensure accurate and complete aggregations?
- A
Set a watermark to allow late data for up to 5 minutes and use windowing with allowed lateness.
- B
Use a sliding window of 5 minutes instead of a fixed-time window to account for late data.
- C
Discard any late-arriving data to simplify the processing logic and ensure low latency.
- D
Set up a global window instead of fixed-time windows to accommodate late-arriving data.
Show answer and explanation
Correct answer: A
Explanation
To handle late-arriving data while using fixed-time windows, you should configure a watermark to delay the finalization of results for the 1-minute window by up to 5 minutes. This allows late events to be included in the aggregation without significantly delaying overall processing. Other options either do not meet the fixed-time window requirement or compromise accuracy.
- A. Correct.
Setting a watermark to allow late data for up to 5 minutes and configuring windowing with allowed lateness ensures that late-arriving data is included in the 1-minute aggregations while maintaining a fixed-time window structure.
- B. Incorrect.
Using a sliding window of 5 minutes does not directly address the problem of late-arriving data for fixed-time windows, as it changes the windowing strategy and may lead to overlapping results.
- C. Incorrect.
Discarding late-arriving data simplifies processing but sacrifices accuracy and completeness of the aggregations, which is not suitable for this use case.
- D. Incorrect.
Using a global window would accept all late data, but it removes the fixed-time windowing requirement and would not provide the desired per-minute aggregations.