Google Professional Data Engineer Question 107
Single answerGoogle Cloud PlatformYou are working as a Data Engineer for a retail company that wants to import daily sales data from multiple on-premises databases into Google BigQuery for analytics. The data needs to be loaded incrementally every night, and the company wants to minimize operational overhead. Which of the following approaches would best meet the requirements?
- A
Use a scheduled Cloud Dataflow pipeline to extract data from the on-premises databases and write it to BigQuery.
- B
Manually export data from the on-premises databases as CSV files and import them into BigQuery using the BigQuery Web UI.
- C
Set up a scheduled transfer using BigQuery Data Transfer Service to pull data directly from the on-premises databases.
- D
Use Google Cloud Storage as a staging area: export data from on-premises databases to Cloud Storage, then use BigQuery's load job to import the data.
Show answer and explanation
Correct answer: A
Explanation
Using Cloud Dataflow to load data incrementally into BigQuery is the most suitable approach because it is a fully managed, scalable ETL solution that can handle the requirements of automation, incremental loads, and minimizing operational overhead. Other options either require manual intervention or involve additional complexity.
- A. Correct.
This is the correct answer. Cloud Dataflow allows you to create scalable, managed pipelines for extracting, transforming, and loading (ETL) data. It supports incremental data loads, can be scheduled, and minimizes operational overhead by automating the process.
- B. Incorrect.
This option is not ideal because it requires manual intervention daily, which increases operational overhead, making it unsuitable for an automated and scalable solution.
- C. Incorrect.
BigQuery Data Transfer Service is not designed to pull data directly from on-premises databases. It is mainly used for transferring data from SaaS applications and Google services.
- D. Incorrect.
While this approach could work, it involves multiple steps (exporting data to Cloud Storage and then loading it into BigQuery), which increases complexity compared to using a single tool like Cloud Dataflow.