Google Professional Data Engineer Question 114
Single answerGoogle Cloud PlatformYou are working as a Data Engineer for a retail company. Your team processes daily sales and inventory data using a workflow that involves ingesting data into BigQuery, transforming it using Dataflow, and finally triggering a machine learning model hosted on AI Platform. The workflow must run automatically every night at 2 AM, and any failures should trigger an alert. Which tool should you use to orchestrate this workflow efficiently?
- A
Cloud Composer
- B
Cloud Scheduler
- C
Dataflow
- D
Workflows
Show answer and explanation
Correct answer: A
Explanation
The described scenario involves a complex workflow with interdependent tasks across multiple Google Cloud services. Cloud Composer is the best choice as it is designed for orchestrating such workflows, providing features like dependency management, scheduling, and alerting for failures. Other options like Cloud Scheduler and Workflows lack the required capabilities for managing complex interdependent workflows.
- A. Correct.
Cloud Composer is a fully managed orchestration service based on Apache Airflow. It is designed for complex workflows with multiple interdependent tasks, making it the best choice for orchestrating the described workflow.
- B. Incorrect.
Cloud Scheduler is suitable for triggering simple tasks at specific times, but it does not provide the capability to manage complex workflows with dependencies.
- C. Incorrect.
Dataflow is a data processing service, not an orchestration tool. While it can handle data transformation tasks, it cannot orchestrate multi-step workflows involving other services.
- D. Incorrect.
Workflows is a service for orchestrating Google Cloud services but is better suited for lightweight and serverless workflows. Cloud Composer is more appropriate for the described scenario due to its support for complex dependencies and monitoring.