Google Professional Machine Learning Engineer Question 97
Select 2Google Cloud PlatformYou are building a machine learning pipeline to process and analyze large datasets for a retail company. The raw data is stored in a Cloud Storage bucket, and you need to perform distributed data transformations and aggregations before storing the processed data into a relational database for downstream analytics. The processed data should also be highly available for real-time querying. Which combination of tools would be the most appropriate to implement this pipeline?
- A
Use Apache Spark to process and transform the data, and store the processed data in Cloud Spanner
- B
Use Apache Hadoop to process and transform the data, and store the processed data in Cloud SQL
- C
Use Apache Spark to process and transform the data, and store the processed data in Cloud SQL
- D
Use Cloud SQL to process and transform the data, and store the processed data in Apache Hadoop
- E
Use Apache Hadoop to process and transform the data, and store the processed data in Cloud Spanner
Show answer and explanation
Correct answers: A, C
Explanation
Apache Spark is well-suited for distributed data processing and can handle large-scale data transformations efficiently. Cloud Spanner is a highly available and scalable relational database, making it ideal for real-time querying scenarios. Alternatively, Cloud SQL can be used for smaller-scale relational data storage when real-time availability and scalability are not critical. Apache Hadoop is better suited for batch processing but lacks the flexibility for real-time data querying, making it less appropriate for this use case.
- A. Correct.
Correct: Apache Spark is a distributed data processing engine suitable for performing large-scale data transformations, and Cloud Spanner provides high availability and scalability for real-time querying and storing processed data.
- B. Incorrect.
Incorrect: While Apache Hadoop can handle distributed processing, Cloud SQL is not designed for horizontally scalable, high availability requirements needed for real-time querying.
- C. Correct.
Correct: Apache Spark is widely used for distributed data processing, and Cloud SQL is a cost-effective option for storing processed data when high availability and scalability are not strict requirements.
- D. Incorrect.
Incorrect: Cloud SQL is not designed for distributed data processing, and Apache Hadoop is primarily used for batch data processing, not for serving processed data in real-time.
- E. Incorrect.
Incorrect: Apache Hadoop is suitable for batch processing, but Cloud Spanner is more appropriate for real-time querying and high availability requirements, which are not typically paired together.