Google Professional Data Engineer Question 71
Select 2Google Cloud PlatformYou are designing a data pipeline on Google Cloud to process streaming data from IoT devices. The data needs to be ingested into BigQuery for real-time analytics, and occasionally archived to Cloud Storage for compliance purposes. Which combination of data sources and sinks would be most appropriate for this scenario?
- A
Use Pub/Sub as the data source and BigQuery as the sink for real-time analytics.
- B
Use Cloud Storage as the data source and BigQuery as the sink for real-time analytics.
- C
Use Pub/Sub as the data source and Cloud Storage as the sink for archiving data.
- D
Use BigQuery as the data source and Pub/Sub as the sink for archiving data.
- E
Use Cloud Storage as the data sink for real-time analytics.
Show answer and explanation
Correct answers: A, C
Explanation
In this scenario, the best design leverages Pub/Sub as the data source for streaming IoT data, as it is highly scalable and supports real-time ingestion. BigQuery is an excellent sink for real-time analytics due to its ability to handle streaming inserts and perform fast SQL-based analytics. For archiving purposes, Cloud Storage is a cost-effective and durable option for storing historical or compliance-related data.
- A. Correct.
Pub/Sub is ideal for ingesting streaming data. BigQuery is a suitable sink for real-time analytics as it supports streaming inserts.
- B. Incorrect.
Cloud Storage is not typically used as a data source for real-time streaming pipelines due to its batch-processing nature.
- C. Correct.
Pub/Sub can handle streaming data ingestion, and Cloud Storage is a good sink for archiving data because it is cost-effective and designed for long-term storage.
- D. Incorrect.
BigQuery is not commonly used as a source for data pipelines. Pub/Sub is also not a suitable sink for data storage or archiving.
- E. Incorrect.
Cloud Storage is not designed for real-time analytics; it is better suited for long-term storage or batch processing.