Google Professional Data Engineer Question 73
Select 3Google Cloud PlatformYour organization has implemented a data pipeline using Google Cloud Dataflow to process streaming data from IoT sensors. The team wants to enrich this data by joining it with a static dataset stored in BigQuery, and then apply custom business logic to transform the data before writing it to a Cloud Storage bucket for downstream analytics. Which of the following steps are required to define and implement the data transformation logic in this scenario?
- A
Write a Dataflow pipeline using Apache Beam to define the transformation steps.
- B
Use BigQuery's SQL interface to define the transformation logic directly in the pipeline.
- C
Leverage Cloud Dataflow's ParDo transform to apply custom business logic.
- D
Use BigQuery to pre-join the static dataset with the streaming data before ingesting it into Dataflow.
- E
Use a combination of GroupByKey and Combine transforms in Dataflow to handle aggregations.
Show answer and explanation
Correct answers: A, C, E
Explanation
To define and implement data transformation logic in a Dataflow pipeline, you need to use Apache Beam's programming model and its transforms. Specifically, you use ParDo for custom logic and Combine/GroupByKey for aggregations. BigQuery can be used as a source or target but cannot define transformations directly within the pipeline.
- A. Correct.
Correct: Apache Beam is the programming model used to define Dataflow pipelines, and it allows you to specify the transformation steps required for processing the data.
- B. Incorrect.
Incorrect: BigQuery's SQL interface cannot be directly used within a Dataflow pipeline. However, BigQuery can serve as a data source or target for the pipeline.
- C. Correct.
Correct: The ParDo transform in Dataflow is used to apply custom logic to each element in the pipeline, which is essential for implementing business-specific transformations.
- D. Incorrect.
Incorrect: While BigQuery can be used for pre-joining data, the question specifies that the join happens within the Dataflow pipeline, so this step is not applicable in this context.
- E. Correct.
Correct: GroupByKey and Combine transforms are commonly used in Dataflow pipelines to perform aggregations, which is likely part of the transformation logic.