Google Professional Data Engineer Question 91
Select 3Google Cloud PlatformYou are working on a data pipeline in Google Cloud that processes streaming data from IoT sensors. The pipeline ingests data into Cloud Pub/Sub and then uses Dataflow for transformations. The raw data contains fields for 'sensor_id', 'timestamp', 'temperature', and 'humidity'. Your task is to transform the data to calculate the hourly average temperature for each sensor. Which of the following transformations in Dataflow are necessary to achieve your goal?
- A
Use a GroupByKey transformation to group the data by 'sensor_id'.
- B
Apply a Windowing function to create hourly time windows.
- C
Use a Filter transformation to exclude data with null 'temperature' values.
- D
Perform a CombinePerKey transformation to calculate the average temperature for each 'sensor_id' within the time window.
- E
Use a Flatten transformation to merge multiple PCollections into a single PCollection.
Show answer and explanation
Correct answers: B, C, D
Explanation
To calculate the hourly average temperature for each sensor, the data must first be divided into hourly time windows using a Windowing function. Filtering out invalid data (e.g., null temperature values) ensures data quality. Afterward, a CombinePerKey transformation is used to efficiently calculate the average temperature for each sensor within the defined time window. GroupByKey and Flatten are not appropriate for the described task.
- A. Incorrect.
GroupByKey is not the correct choice here because GroupByKey is not efficient for aggregation operations like calculating averages. More suitable options exist for this scenario, such as CombinePerKey.
- B. Correct.
Windowing is necessary to group the data into hourly time intervals, which is a requirement for calculating the hourly average temperature.
- C. Correct.
Filtering out null temperature values ensures data quality and prevents errors during the aggregation step.
- D. Correct.
CombinePerKey is the appropriate transformation to calculate averages for each 'sensor_id' within the hourly time windows.
- E. Incorrect.
Flatten is not needed in this scenario as there is no mention of merging multiple PCollections. The focus is on transforming a single stream of data.