Google Professional Data Engineer Question 103
Single answerGoogle Cloud PlatformYou are a data engineer working for a retail company. The marketing team has requested a one-time ingestion of a large CSV file containing customer feedback data into BigQuery for analysis. The file is currently stored in a Cloud Storage bucket. What is the most efficient and cost-effective method to handle this ad hoc ingestion?
- A
Use the BigQuery Data Transfer Service to schedule a transfer from Cloud Storage.
- B
Manually upload the CSV file into BigQuery using the BigQuery Console.
- C
Write a Dataflow pipeline to read the file from Cloud Storage and write it to BigQuery.
- D
Use the bq command-line tool to load the file directly from Cloud Storage into BigQuery.
Show answer and explanation
Correct answer: D
Explanation
For one-time ingestion of a file from Cloud Storage into BigQuery, the bq command-line tool provides a straightforward, efficient, and cost-effective solution. It is specifically designed for tasks like loading data directly into BigQuery without the need for complex pipelines or recurring schedules. The other options either involve unnecessary complexity, are not suitable for ad hoc ingestion, or are less efficient for large files.
- A. Incorrect.
The BigQuery Data Transfer Service is intended for automated, recurring data transfers and does not support ad hoc ingestion of files from Cloud Storage.
- B. Incorrect.
Manually uploading the file via the BigQuery Console is possible, but it is less efficient and not suitable for large files compared to using the command-line tool.
- C. Incorrect.
While Dataflow can handle complex data processing and ingestion, it is overkill for a simple one-time ingestion and incurs additional cost and complexity.
- D. Correct.
The bq command-line tool is the most efficient and cost-effective method for ad hoc ingestion as it supports loading files directly from Cloud Storage into BigQuery with minimal setup.