Google Professional Data Engineer Question 233
Select 3Google Cloud PlatformYou are designing a data pipeline on Google Cloud to process daily sales data from multiple retail stores. The pipeline must handle the following requirements:
- The data from all stores must be ingested automatically every day without manual intervention.
- The solution must allow for reprocessing historical data in case of failures or updates to the processing logic.
- The pipeline should be reusable across multiple regions with minimal changes.
Which combination of approaches should you use to meet these requirements?
- A
Use Cloud Composer to schedule daily data ingestion tasks and configure it to trigger Dataflow jobs for processing.
- B
Store raw data in Cloud Storage with a consistent naming convention for folders and files to support reprocessing.
- C
Create a custom script to manually run the pipeline whenever historical data needs to be reprocessed.
- D
Use Terraform or Deployment Manager to provision the pipeline infrastructure, ensuring consistent deployments across regions.
- E
Hardcode regional configurations into the pipeline code to ensure it works for specific locations.
Show answer and explanation
Correct answers: A, B, D
Explanation
The correct combination of approaches ensures automation, reusability, and support for historical data reprocessing. Cloud Composer provides orchestration for daily ingestion and processing, Cloud Storage with consistent naming enables data reprocessing, and using Terraform or Deployment Manager ensures repeatable deployments across regions. Avoiding manual processes and hardcoding configurations aligns with best practices for automation and repeatability in Google Cloud.
- A. Correct.
Using Cloud Composer for scheduling ensures automation and allows you to orchestrate workflows like triggering Dataflow jobs. This supports repeatability and reduces manual intervention.
- B. Correct.
Storing raw data in Cloud Storage with a consistent naming convention allows for easy reprocessing of historical data by referencing specific files or folders. This design enables flexibility and repeatability.
- C. Incorrect.
Manually running pipelines for historical data reprocessing contradicts the requirement for automation and repeatability. This is not a scalable or reliable approach.
- D. Correct.
Using Terraform or Deployment Manager to manage infrastructure ensures consistent deployments across regions and makes the pipeline reusable with minimal changes.
- E. Incorrect.
Hardcoding regional configurations limits the flexibility and reusability of the pipeline, making it less suitable for multi-region deployments.