Google Professional Data Engineer Question 106
Select 2Google Cloud PlatformYou are working for a retail company that collects sales data from its stores every minute. The data is stored in CSV files in an on-premises server. The company wants to analyze this data in real-time using Google BigQuery. Which of the following approaches will allow you to ingest the data into BigQuery efficiently and in near real-time?
- A
Use Cloud Pub/Sub to stream the data into BigQuery after sending the CSV files to a Cloud Storage bucket.
- B
Set up a Dataflow pipeline to read the CSV files from the on-premises server and stream the data into BigQuery.
- C
Use BigQuery's batch loading feature to upload the CSV files once every 24 hours from the on-premises server.
- D
Set up a Transfer Appliance to move the CSV files to Google Cloud Storage and then load them into BigQuery.
- E
Use a third-party tool to replicate the data from the on-premises server directly into BigQuery in real-time.
Show answer and explanation
Correct answers: A, B
Explanation
To achieve real-time or near real-time data ingestion into BigQuery, Cloud Pub/Sub and Dataflow are the best options. Pub/Sub enables streaming data pipelines, and Dataflow provides a fully managed service for streaming data processing. Batch loading and offline migration tools like Transfer Appliance are not appropriate for real-time requirements. Native Google Cloud tools are preferred over third-party tools for reliability and integration.
- A. Correct.
This approach is valid because Cloud Pub/Sub can be used as a messaging service to stream data into BigQuery in real-time. However, additional processing might be required to parse the CSV files.
- B. Correct.
Using Dataflow is a suitable approach for real-time or near real-time data ingestion. Dataflow can process and stream the data from the on-premises server into BigQuery efficiently.
- C. Incorrect.
BigQuery's batch loading feature is not suitable for real-time or near real-time ingestion as it involves uploading data in batches, which introduces delays.
- D. Incorrect.
Transfer Appliance is designed for large-scale offline data migration rather than real-time data ingestion. This approach is not suitable for continuous real-time data analysis.
- E. Incorrect.
While a third-party tool might allow real-time replication, this option is not recommended as it introduces dependency on external systems and is less efficient compared to native Google Cloud services like Pub/Sub and Dataflow.