Google Professional Data Engineer Question 232
Select 3Google Cloud PlatformYour team is responsible for setting up a data pipeline on Google Cloud to process daily sales data from multiple regions. The pipeline needs to be scalable, fault-tolerant, and reusable, as new regions will be added over time. Which of the following design choices should you implement to ensure automation and repeatability in the pipeline setup?
- A
Use Infrastructure as Code (IaC) tools like Terraform to define and deploy pipeline resources.
- B
Manually configure each pipeline component in the Google Cloud Console for fine-grained control.
- C
Implement a CI/CD pipeline to automate deployment and testing of pipeline code changes.
- D
Store pipeline configuration and metadata in an external, version-controlled repository.
- E
Hardcode region-specific parameters directly into the pipeline code for easier management.
Show answer and explanation
Correct answers: A, C, D
Explanation
To design automation and repeatability in a data pipeline, you should focus on using tools and practices that support declarative resource management, automated deployments, and maintainable configurations. Terraform automates infrastructure provisioning, CI/CD pipelines streamline code deployment and testing, and version control systems provide a centralized and trackable way to manage configuration. Avoid manual processes and hardcoding to ensure the pipeline is scalable and maintainable.
- A. Correct.
Using Infrastructure as Code (IaC) tools like Terraform ensures that resources are defined declaratively and can be consistently deployed, making the pipeline setup repeatable and scalable.
- B. Incorrect.
Manually configuring components in the Google Cloud Console is time-consuming, error-prone, and not easily repeatable, which makes it unsuitable for automation and scalability.
- C. Correct.
A CI/CD pipeline enables automated delivery and testing of changes, reducing the risk of errors and ensuring repeatable deployments of the pipeline.
- D. Correct.
Storing configuration and metadata in an external repository allows for version control and promotes collaboration, making the pipeline setup more repeatable and maintainable.
- E. Incorrect.
Hardcoding region-specific parameters makes the pipeline less flexible and reusable, as adding new regions would require manual code changes.