Google Professional Data Engineer Question 89
Select 3Google Cloud PlatformA retail company wants to create a data pipeline to process their real-time transactional data and store it in a data warehouse for analytics. The pipeline should include real-time ingestion, transformations, and the ability to query the data using SQL. Which combination of Google Cloud services would best accomplish this task?
- A
Pub/Sub for real-time messaging and BigQuery for data warehousing
- B
Dataflow for data transformations and BigQuery for data warehousing
- C
Cloud Data Fusion for real-time ingestion and Hadoop for data warehousing
- D
Dataproc for batch processing and Pub/Sub for real-time data ingestion
- E
Pub/Sub for real-time messaging, Dataflow for transformations, and BigQuery for data warehousing
Show answer and explanation
Correct answers: A, B, E
Explanation
To build a real-time data pipeline for transactional data, you need services that handle ingestion, transformations, and storage for analytics. Pub/Sub is used for real-time messaging, Dataflow for processing and transformations, and BigQuery as the data warehouse for SQL-based queries. The combination of these services provides a robust, scalable, and efficient solution for the scenario.
- A. Correct.
Pub/Sub is ideal for ingesting real-time transactional data, and BigQuery serves as an effective data warehouse for analytics. However, this option does not address data transformation needs, which are crucial in this pipeline.
- B. Correct.
Dataflow is excellent for transformation tasks, and BigQuery is suitable for data warehousing. However, this option does not specify a real-time ingestion service like Pub/Sub.
- C. Incorrect.
Cloud Data Fusion is more geared towards batch data integration and does not provide real-time ingestion capabilities. Hadoop is not a native Google Cloud service and is less optimized for a modern analytics workflow compared to BigQuery.
- D. Incorrect.
Dataproc is typically used for batch processing, not real-time processing. Pub/Sub is valid for real-time ingestion, but the combination does not include a data warehouse for analytics.
- E. Correct.
This combination is the most comprehensive. Pub/Sub handles real-time messaging, Dataflow enables transformations, and BigQuery serves as the data warehouse, addressing all requirements of the pipeline.