Google Professional Data Engineer Question 72
Select 2Google Cloud PlatformYou are tasked with designing a data pipeline on Google Cloud for a company that processes clickstream data. The data originates from an on-premises system and must be ingested into Google Cloud for analysis. The data should be stored in a format optimized for querying using BigQuery, and the pipeline must support both batch and streaming ingestion. Which combination of data sources and sinks would best meet these requirements?
- A
Set up Cloud Pub/Sub as the data source and BigQuery as the sink.
- B
Set up a Cloud Storage bucket as the data source and BigQuery as the sink.
- C
Use Cloud Dataflow to read from Cloud Pub/Sub and write to BigQuery.
- D
Use Apache Kafka as the data source and Cloud Spanner as the sink.
- E
Configure Transfer Appliance to ingest the data and write it directly to BigQuery.
Show answer and explanation
Correct answers: A, C
Explanation
To meet the requirements of supporting both batch and streaming ingestion, the optimal approach is to use Cloud Pub/Sub as the data source for streaming and BigQuery as the sink for querying. Cloud Dataflow can be used to process data from Pub/Sub and write it to BigQuery, ensuring compatibility with both ingestion modes. Other options either lack streaming support or introduce unnecessary complexity.
- A. Correct.
Cloud Pub/Sub is a managed messaging service ideal for streaming ingestion. BigQuery is optimized for querying large datasets, and this combination supports both streaming and batch ingestion when paired with a processing tool like Cloud Dataflow.
- B. Incorrect.
While Cloud Storage can serve as a data source, it is better suited for batch ingestion rather than handling real-time streaming data. Therefore, it is not the most optimal choice for this scenario.
- C. Correct.
Cloud Dataflow is a powerful tool for real-time and batch data processing. It can integrate with Cloud Pub/Sub as a source and BigQuery as a sink, making it an ideal choice for this pipeline.
- D. Incorrect.
Apache Kafka is not a native Google Cloud service. While it can be configured to work with Google Cloud, it introduces complexity and is not the most straightforward approach for this requirement.
- E. Incorrect.
Transfer Appliance is designed for bulk data transfer from on-premises to Cloud Storage or BigQuery. It is not suitable for streaming ingestion and is therefore not appropriate for this use case.