Google Professional Data Engineer Question 82
Select 2Google Cloud PlatformYou are designing a data pipeline in Google Cloud to process real-time clickstream data from a high-traffic e-commerce website. The data must be ingested, processed in near-real-time, and stored for both analytics and reporting. Which combination of services would best meet these requirements?
- A
Use Pub/Sub for ingestion, Dataflow for processing, and BigQuery for storage.
- B
Use Cloud Storage for ingestion, Dataproc for processing, and Cloud SQL for storage.
- C
Use Pub/Sub for ingestion, Dataproc for batch processing, and Bigtable for storage.
- D
Use Pub/Sub for ingestion, Dataflow for processing, and Cloud Storage for archival storage.
Show answer and explanation
Correct answers: A, D
Explanation
To build a real-time data pipeline, Pub/Sub is a reliable service for ingesting event streams, and Dataflow provides robust support for stream processing. For storing processed data, BigQuery is optimal for analytics and reporting, while Cloud Storage can be used for long-term archival. Depending on the use case, a combination of these services ensures scalability, low latency, and analytical capabilities.
- A. Correct.
Correct: Pub/Sub is well-suited for real-time ingestion, Dataflow supports stream processing, and BigQuery is an excellent choice for analytics and reporting.
- B. Incorrect.
Incorrect: Cloud Storage is not ideal for real-time ingestion, and Cloud SQL is not optimized for analytics or high-throughput storage.
- C. Incorrect.
Incorrect: While Pub/Sub can handle ingestion, Dataproc is more suitable for batch processing rather than real-time, and Bigtable is not designed for analytics or reporting.
- D. Correct.
Correct: Pub/Sub for ingestion and Dataflow for stream processing match the real-time requirement, while Cloud Storage can be used for archival purposes but not analytics.