Google Professional Data Engineer Question 179
Single answerGoogle Cloud PlatformYou are a data engineer tasked with creating a pipeline that integrates data from multiple sources and sends it to BigQuery for analytics. You want to use a managed service to orchestrate this workflow and also ensure it seamlessly connects with other tools like Cloud Storage and Pub/Sub. Which service should you choose?
- A
Cloud Functions
- B
Cloud Dataflow
- C
Cloud Composer
- D
Dataproc
Show answer and explanation
Correct answer: C
Explanation
Cloud Composer is the correct choice because it is specifically designed for orchestrating workflows across various tools and services in Google Cloud, such as Cloud Storage, Pub/Sub, and BigQuery. It provides a scalable, managed solution to build, schedule, and monitor workflows, making it the best fit for the given scenario.
- A. Incorrect.
Cloud Functions is a serverless compute service used for event-driven applications. While it can handle small tasks, it is not well-suited for orchestrating complex workflows involving multiple tools.
- B. Incorrect.
Cloud Dataflow is a stream and batch data processing service, but it is not specifically designed for orchestration across multiple tools.
- C. Correct.
Cloud Composer is a managed workflow orchestration service based on Apache Airflow. It is ideal for connecting and managing pipelines that involve multiple tools like Cloud Storage, Pub/Sub, and BigQuery.
- D. Incorrect.
Dataproc is a managed service for running Apache Spark and Hadoop clusters but is not intended for orchestrating workflows across multiple tools.