Google Professional Data Engineer Question 105
Select 2Google Cloud PlatformYou are working for a retail company that needs to ingest and process large volumes of sales data from multiple on-premises databases into Google Cloud for real-time analytics. Which of the following approaches would be most appropriate to achieve this in a scalable and efficient manner?
- A
Use Cloud Storage as a staging area and transfer data to BigQuery using scheduled batch jobs.
- B
Set up a Pub/Sub topic to stream data from the on-premises databases directly into BigQuery.
- C
Leverage Dataflow to implement a streaming pipeline that ingests data from on-premises databases and writes it to BigQuery.
- D
Manually export the data from on-premises databases, upload it to Cloud Storage, and then import it into BigQuery using the web UI.
- E
Deploy Transfer Appliance to physically ship data to Google Cloud and import the data into BigQuery.
Show answer and explanation
Correct answers: B, C
Explanation
For real-time analytics, a combination of Google Cloud services like Pub/Sub and Dataflow is ideal. Pub/Sub allows real-time ingestion of data from on-premises databases, while Dataflow can process and transform the streaming data before loading it into BigQuery. These services ensure scalability and low latency, unlike batch processing or manual methods which are not suitable for real-time needs.
- A. Incorrect.
Using Cloud Storage as a staging area and scheduling batch jobs is suitable for batch processing but does not fulfill the real-time analytics requirement.
- B. Correct.
Using a Pub/Sub topic for streaming data directly into BigQuery is a scalable and efficient solution for real-time data ingestion.
- C. Correct.
Dataflow is ideal for building streaming pipelines and allows you to process and transform data in real-time before loading it into BigQuery.
- D. Incorrect.
Manually exporting, uploading, and importing data is not scalable or efficient for real-time analytics.
- E. Incorrect.
Transfer Appliance is designed for bulk data transfers and is not suitable for real-time ingestion or streaming scenarios.