Google Professional Data Engineer Question 234
Select 3Google Cloud PlatformYour team is responsible for building a data pipeline in Google Cloud that ingests, transforms, and stores data for multiple departments. You want to ensure that the pipeline is automated, scalable, and repeatable while minimizing manual intervention. Which of the following design choices will help you achieve this goal?
- A
Use Cloud Composer to orchestrate pipeline tasks with DAGs.
- B
Write custom scripts to manually trigger data processing jobs as needed.
- C
Implement Infrastructure as Code (IaC) solutions like Terraform to provision pipeline resources.
- D
Use dynamic parameters in Dataflow templates to handle different processing requirements.
- E
Manually deploy pipeline updates to ensure precise control over changes.
Show answer and explanation
Correct answers: A, C, D
Explanation
To design an automated and repeatable data pipeline, it is essential to use tools and strategies that minimize manual effort, ensure consistency, and support scalability. Cloud Composer provides workflow orchestration, IaC tools like Terraform automate resource provisioning, and dynamic Dataflow templates ensure reusability and adaptability. Avoiding manual processes, such as triggering jobs or deploying updates, is critical to achieving these objectives.
- A. Correct.
Using Cloud Composer with Directed Acyclic Graphs (DAGs) is a best practice for orchestrating and automating complex workflows, making the pipeline repeatable and minimizing manual intervention.
- B. Incorrect.
Writing custom scripts for manual triggering is not scalable or automated and goes against the goal of repeatability.
- C. Correct.
Implementing Infrastructure as Code (IaC) tools like Terraform ensures that provisioning is automated and consistent, which supports repeatability and scalability.
- D. Correct.
Dynamic parameters in Dataflow templates allow for reusability and flexibility, making the pipeline more automated and reducing the need for manually creating separate jobs for different requirements.
- E. Incorrect.
Manually deploying updates is error-prone and not automated, which contradicts the goal of designing a repeatable and automated system.