Google Professional Data Engineer Question 110
Select 3Google Cloud PlatformYou are a data engineer at a retail company. Your team wants to integrate an external e-commerce platform's API into your Google Cloud environment to analyze sales data. The API provides data in JSON format and updates every hour. Which combination of tools and techniques should you use to efficiently ingest and process this data?
- A
Use Cloud Functions to trigger on an hourly schedule, fetch the data, and write it to a Cloud Storage bucket.
- B
Use Pub/Sub to capture API responses and trigger downstream processing pipelines.
- C
Use Cloud Dataflow to parse the JSON data from the API and write it directly to BigQuery.
- D
Use Cloud Scheduler to trigger a custom script hosted on Compute Engine to fetch and process the data.
- E
Use BigQuery's federated queries to directly query the API endpoint without storing the data.
Show answer and explanation
Correct answers: A, B, C
Explanation
Integrating with new data sources, such as an external API, requires a combination of tools that can handle data ingestion, processing, and storage. Cloud Functions can fetch data on a schedule and store it in Cloud Storage. Pub/Sub can decouple data ingestion and processing, making the system scalable. Cloud Dataflow is an excellent tool to parse and transform JSON data before storing it in BigQuery. The other options are either less efficient or not supported for this use case.
- A. Correct.
This is a valid option. Cloud Functions can be triggered on a schedule (via Cloud Scheduler) to fetch the API data and store it in Cloud Storage for further processing.
- B. Correct.
This is a valid option. Pub/Sub can be used to decouple and manage API responses, allowing for scalability and triggering downstream processing pipelines.
- C. Correct.
This is a valid option. Cloud Dataflow can process and transform JSON data in a streaming or batch mode and load it into BigQuery for analysis.
- D. Incorrect.
While technically possible, this approach is less efficient and scalable compared to serverless options like Cloud Functions or Dataflow. It also requires managing infrastructure, which increases operational overhead.
- E. Incorrect.
BigQuery does not support direct querying of API endpoints. Data must first be ingested into a supported storage or database format before querying.