Google Professional Data Engineer Question 102
Single answerGoogle Cloud PlatformYou are tasked with building a data ingestion pipeline in Google Cloud for a one-time batch process. The data is stored in CSV files on a remote FTP server, and you need to transfer the files to BigQuery for analysis. The solution should ensure minimal setup and leverage Google Cloud services effectively. Which approach should you choose?
- A
Use Cloud Storage Transfer Service to directly transfer the data from the FTP server to BigQuery.
- B
Download the files locally, upload them to Cloud Storage, and then use BigQuery's import feature to load the data.
- C
Write a custom Python script using the BigQuery API to fetch data from the FTP server and load it into BigQuery.
- D
Use a Cloud Function triggered by a Pub/Sub message to retrieve the data from the FTP server, save it to Cloud Storage, and then load it into BigQuery.
Show answer and explanation
Correct answer: B
Explanation
For a one-time batch ingestion process, manually downloading the data, uploading it to Cloud Storage, and using BigQuery's import feature is the simplest and most efficient approach. It avoids unnecessary complexity and leverages BigQuery's built-in capabilities for data ingestion.
- A. Incorrect.
Cloud Storage Transfer Service does not support direct transfers from FTP servers. It is designed for transferring data from supported cloud providers or HTTP/HTTPS endpoints to Cloud Storage.
- B. Correct.
This is a valid approach for a one-time batch process. Manually downloading the files locally, uploading them to Cloud Storage, and using BigQuery's import feature is straightforward and requires minimal setup for a one-time task.
- C. Incorrect.
While it is possible to write a custom Python script using the BigQuery API, this approach requires significant effort and is not recommended for a one-time batch process unless absolutely necessary.
- D. Incorrect.
Using a Cloud Function and Pub/Sub is better suited for automated or recurring ingestion pipelines. It introduces unnecessary complexity for a one-time batch process.