Google Professional Data Engineer Question 104
Single answerGoogle Cloud PlatformYou are designing a data pipeline to ingest a one-time batch of data from an on-premises MySQL database into Google BigQuery for analysis. The data volume is around 500 GB, and you want to minimize manual intervention while ensuring reliability. Which solution should you choose?
- A
Use the BigQuery Data Transfer Service to directly connect to the MySQL database and import the data.
- B
Export the MySQL data to CSV files, upload them to a Cloud Storage bucket, and then load them into BigQuery using the BigQuery Data Transfer Service.
- C
Export the MySQL data to CSV files, upload them to a Cloud Storage bucket, and use the BigQuery Data Loading feature to import the data.
- D
Set up a Dataflow pipeline to stream the data from MySQL into BigQuery in real-time.
Show answer and explanation
Correct answer: C
Explanation
The most suitable solution for a one-time batch ingestion of 500 GB of data from MySQL to BigQuery involves exporting the data to CSV files, uploading them to a Cloud Storage bucket, and then using the BigQuery Data Loading feature. This approach ensures reliability, minimizes manual intervention, and leverages scalable tools. Other options either involve unsupported tools (e.g., BigQuery Data Transfer Service) or add unnecessary complexity (e.g., real-time Dataflow pipeline).
- A. Incorrect.
The BigQuery Data Transfer Service is not suitable for ingesting data directly from MySQL databases. It is designed for pre-built connectors like Google Ads or Google Analytics.
- B. Incorrect.
The BigQuery Data Transfer Service does not support importing data from CSV files. This option is not feasible for this scenario.
- C. Correct.
This is the correct approach. Exporting the data to CSV files, uploading them to a Cloud Storage bucket, and then using the BigQuery Data Loading feature ensures reliability and scalability for a one-time batch ingestion.
- D. Incorrect.
Setting up a Dataflow pipeline for real-time streaming is not necessary for one-time batch data ingestion. It would also add unnecessary complexity and cost.