Google Professional Data Engineer Question 42
Single answerGoogle Cloud PlatformYour company is planning to implement a real-time recommendation system for its e-commerce platform. Current requirements include low-latency processing of user behavior data and providing recommendations within milliseconds of user interaction. The business also anticipates scaling to handle a 10x increase in traffic over the next two years. Which architecture best meets both the current and future requirements?
- A
Use a batch processing system with Cloud Dataflow to process user behavior data and BigQuery for storing recommendation results.
- B
Implement a real-time streaming pipeline using Pub/Sub, Dataflow, and Bigtable to process user behavior data and serve low-latency recommendations.
- C
Deploy a Cloud SQL database for storing and processing user behavior data, combined with a custom API for serving recommendations.
- D
Use a Hadoop-based solution running on Compute Engine to process user behavior data and generate recommendations.
Show answer and explanation
Correct answer: B
Explanation
The correct answer is to implement a real-time streaming pipeline using Pub/Sub, Dataflow, and Bigtable. This architecture is fully managed, scalable, and optimized for real-time, low-latency use cases like recommendation systems. It ensures the company can meet current performance requirements while easily scaling to accommodate future traffic growth.
- A. Incorrect.
Batch processing is not suitable for low-latency requirements as it introduces delays in processing and delivering recommendations. Additionally, BigQuery is optimized for analytical queries, not real-time recommendation serving.
- B. Correct.
Pub/Sub, Dataflow, and Bigtable provide a scalable, low-latency, and real-time streaming architecture. Pub/Sub handles real-time ingestion, Dataflow processes the data, and Bigtable serves as a low-latency database for recommendations. This meets both current low-latency needs and future scalability requirements.
- C. Incorrect.
Cloud SQL is not designed for high-throughput, low-latency streaming workloads. It is better suited for transactional or small-scale workloads and would struggle to handle a 10x increase in traffic effectively.
- D. Incorrect.
Hadoop-based solutions on Compute Engine are typically batch-oriented and not designed for real-time processing. Additionally, they may not scale as efficiently compared to fully managed services like Pub/Sub, Dataflow, and Bigtable.