Google Professional Data Engineer Question 92
Single answerGoogle Cloud PlatformYou are designing a data pipeline in Google Cloud to process large amounts of unstructured log data. The logs need to be parsed, transformed into a structured format, and enriched with additional metadata from a reference dataset stored in BigQuery. Which transformation step should you use in Dataflow to efficiently enrich the log data with metadata?
- A
GroupByKey
- B
Join
- C
Flatten
- D
Filter
Show answer and explanation
Correct answer: B
Explanation
Data enrichment typically involves merging or joining one dataset with another based on a shared key. In this scenario, the Join transformation in Dataflow is the most appropriate choice because it allows you to combine the log data with the metadata stored in BigQuery. Other transformations like GroupByKey, Flatten, and Filter do not fulfill the requirements of this task.
- A. Incorrect.
GroupByKey is used for aggregating data based on a key, but it is not suitable for enriching data from a reference dataset.
- B. Correct.
Join is the correct choice for combining the log data with metadata from the reference dataset based on a shared key.
- C. Incorrect.
Flatten is used to merge multiple PCollections into a single PCollection and is not applicable for enriching or joining datasets.
- D. Incorrect.
Filter is used to remove elements from a dataset based on a condition, not for enriching data with additional metadata.