Google Professional Data Engineer Question 116
Single answerGoogle Cloud PlatformYou are a data engineer at a retail company, and you need to automate a daily ETL workflow that processes sales data stored in Cloud Storage and loads it into BigQuery. The workflow involves running a Dataflow job, performing a data quality check using a Python script, and sending a success or failure notification to a Slack channel. Which Google Cloud service would be the most appropriate to orchestrate this workflow?
- A
Cloud Functions
- B
Cloud Composer
- C
Cloud Scheduler
- D
Cloud Tasks
Show answer and explanation
Correct answer: B
Explanation
Cloud Composer is best suited for orchestrating complex, multi-step workflows like the one described in the scenario. It allows you to define dependencies between tasks, schedule workflows, and integrate with other Google Cloud services such as Dataflow and BigQuery. The other options are either designed for single tasks or lack the required orchestration capabilities.
- A. Incorrect.
Cloud Functions is a serverless platform for running event-driven code in response to triggers. While it can be used for individual tasks, it is not designed for orchestrating multi-step workflows with dependencies.
- B. Correct.
Cloud Composer is a managed Apache Airflow service that is specifically designed for orchestrating complex workflows, including ETL jobs, data quality checks, and notifications. It supports dependencies and task scheduling, making it the most suitable option for this scenario.
- C. Incorrect.
Cloud Scheduler is a service for scheduling single tasks or jobs at specific intervals. However, it does not support multi-step workflows with task dependencies.
- D. Incorrect.
Cloud Tasks is a service for managing asynchronous tasks and queues. It is not designed for orchestrating complex workflows with task dependencies and scheduling.