Google Professional Data Engineer Question 115
Single answerGoogle Cloud PlatformYour company processes real-time customer data and stores it in BigQuery. You need to automate a pipeline that extracts data from Pub/Sub, processes it with Dataflow, and loads the results into BigQuery. The pipeline must also ensure that downstream tasks, such as daily report generation, are triggered automatically upon successful data load into BigQuery. Which tool should you use to orchestrate and automate this workflow?
- A
Cloud Composer
- B
Cloud Functions
- C
Cloud Run
- D
Cloud Workflows
Show answer and explanation
Correct answer: A
Explanation
Cloud Composer is a managed Apache Airflow service that provides an enterprise-grade solution for building, scheduling, and monitoring workflows. In this scenario, where multiple Google Cloud services need orchestration (Pub/Sub, Dataflow, BigQuery), Cloud Composer is the most suitable option due to its robust capabilities in managing dependencies, task scheduling, and error handling. Other tools like Cloud Functions and Cloud Run lack the required orchestration features, while Cloud Workflows is better for lightweight, event-driven workflows.
- A. Correct.
Cloud Composer is the best choice for orchestrating data pipelines and automating complex workflows across services like Pub/Sub, Dataflow, and BigQuery. It provides robust scheduling, monitoring, and dependency management.
- B. Incorrect.
Cloud Functions is a lightweight, serverless compute service best suited for single-purpose tasks. It lacks the orchestration capabilities required for managing complex workflows.
- C. Incorrect.
Cloud Run is a serverless container orchestration service for running stateless applications. It is not designed to handle workflow orchestration across multiple services.
- D. Incorrect.
Cloud Workflows can orchestrate workflows, but it is better suited for lightweight, event-driven workflows. For complex data pipelines with dependencies between tasks, Cloud Composer is more appropriate.