Google Professional Data Engineer Question 149
Select 3Google Cloud PlatformYour company wants to build a data pipeline to process and analyze e-commerce transaction data in near real-time. The business requirements include ensuring high scalability, low latency, fault tolerance, and the ability to generate daily sales reports. As a Data Engineer, what key factors should you consider when mapping these business requirements to a Google Cloud solution?
- A
Choose a managed stream processing service like Dataflow to handle real-time data processing.
- B
Use BigQuery for storing and analyzing large volumes of processed data.
- C
Deploy a self-managed Apache Kafka cluster on Compute Engine instances for data ingestion.
- D
Design the pipeline to use Cloud Storage for real-time transaction processing.
- E
Implement a fault-tolerant architecture with Pub/Sub for data ingestion and message durability.
Show answer and explanation
Correct answers: A, B, E
Explanation
When mapping business requirements to a Google Cloud solution, it is important to choose managed services that align with the requirements of scalability, low latency, fault tolerance, and reporting. Dataflow is ideal for real-time stream processing, BigQuery supports scalable analytics and reporting, and Pub/Sub ensures reliable and fault-tolerant data ingestion. Self-managed solutions like Apache Kafka and inappropriate services like Cloud Storage for real-time processing do not align with the specified requirements.
- A. Correct.
Dataflow is a fully managed stream and batch processing service that is highly scalable and suitable for handling real-time data pipelines. It aligns with the requirement of low latency and fault tolerance.
- B. Correct.
BigQuery is an ideal choice for storing and analyzing large datasets, including daily sales reports. It provides scalability and fast query performance, meeting the reporting requirement.
- C. Incorrect.
While Apache Kafka is a powerful tool for data ingestion, deploying it on Compute Engine requires significant management overhead. A managed service like Pub/Sub is a better fit for scalability and fault tolerance.
- D. Incorrect.
Cloud Storage is not designed for real-time transaction processing. It is more suited for batch processing, archival, or storing raw data.
- E. Correct.
Pub/Sub is a managed message broker that guarantees message durability and supports fault-tolerant ingestion, aligning with the business need for scalability and reliability.