Google Professional Data Engineer Question 108
Select 3Google Cloud PlatformYour company has recently acquired a new data source, which provides real-time event data via a REST API. The data must be ingested into Google Cloud for further processing and analysis. You need to design an architecture that ensures reliable ingestion, supports scalability, and integrates seamlessly with other Google Cloud services. Which of the following components would you use to implement this solution?
- A
Cloud Pub/Sub for real-time event ingestion
- B
Cloud Functions to poll the REST API and publish messages
- C
BigQuery for directly querying the REST API for real-time data
- D
Cloud Storage for storing the raw data from the REST API
- E
Dataflow for processing and transforming the ingested data
Show answer and explanation
Correct answers: A, B, E
Explanation
To integrate with a new data source providing real-time events via a REST API, you need a combination of services that ensure scalable, reliable ingestion and processing. Cloud Functions can poll the REST API and publish the data to Cloud Pub/Sub, which acts as the ingestion layer. Dataflow can then process and transform the data as it flows through the pipeline, enabling seamless integration with downstream analytics or storage services. BigQuery and Cloud Storage are not ideal for real-time ingestion but can be used later in the pipeline for analysis or storage.
- A. Correct.
Cloud Pub/Sub is a fully managed messaging service that is highly suitable for real-time event ingestion. It provides scalability and reliability, making it an essential component in this architecture.
- B. Correct.
Cloud Functions can be used to poll the REST API and publish the incoming data to Cloud Pub/Sub. This serverless option ensures simplicity and automatic scaling for handling incoming events.
- C. Incorrect.
BigQuery is a data warehouse and is not designed for querying data directly from a REST API in real time. It is better suited for analyzing data after it has been ingested and processed.
- D. Incorrect.
Cloud Storage is ideal for storing raw files or data batches but is not suitable for real-time event ingestion from a REST API.
- E. Correct.
Dataflow is a fully managed stream and batch processing service that is well-suited for transforming and enriching the data ingested via Cloud Pub/Sub before further analysis.