Google Professional Machine Learning Engineer Question 239
Select 2Google Cloud PlatformYou are designing a machine learning pipeline on Google Cloud to process data from multiple sources. The data includes CSV files stored in Cloud Storage, JSON files from an external API, and images stored in a Hadoop Distributed File System (HDFS). Which of the following tools/services should you use to efficiently ingest and preprocess this data for machine learning?
- A
Cloud Dataflow
- B
BigQuery Data Transfer Service
- C
TensorFlow Datasets
- D
Apache Beam
- E
Cloud Pub/Sub
Show answer and explanation
Correct answers: A, D
Explanation
To efficiently ingest and preprocess data from multiple sources such as Cloud Storage, APIs, and HDFS, Cloud Dataflow and Apache Beam are the most appropriate tools. Apache Beam provides the unified programming model, while Cloud Dataflow serves as the managed service to execute these pipelines on Google Cloud. Other services like BigQuery Data Transfer Service and Cloud Pub/Sub are not designed for the specified use case, and TensorFlow Datasets is not a data ingestion tool.
- A. Correct.
Cloud Dataflow is a managed service for stream and batch data processing that supports data transformation and ingestion from diverse sources, including Cloud Storage, APIs, and HDFS. It is suitable for preprocessing data for machine learning pipelines.
- B. Incorrect.
BigQuery Data Transfer Service is designed to automate data movement from SaaS applications and analytics services into BigQuery. However, it is not suitable for ingesting and preprocessing data from diverse file types like CSV, JSON, and images.
- C. Incorrect.
TensorFlow Datasets is a library of pre-built datasets for TensorFlow and is not intended for ingestion and preprocessing of raw data from multiple sources.
- D. Correct.
Apache Beam is an open-source unified programming model for defining batch and streaming data-parallel processing pipelines. It is ideal for data ingestion and preprocessing when used with Cloud Dataflow.
- E. Incorrect.
Cloud Pub/Sub is a messaging service designed for real-time event streaming and not intended for preprocessing or ingesting batch data from diverse file sources.