Google Professional Machine Learning Engineer Question 178
Select 2Google Cloud PlatformYou are developing a machine learning pipeline on Google Cloud and need to process and analyze a large dataset using distributed computing. The dataset will undergo feature engineering, and later you need to build a deep learning model. The feature engineering step requires distributed processing, while the deep learning model must be trained on GPUs. Which combination of frameworks should you use to efficiently achieve these tasks?
- A
Apache Spark for distributed processing and TensorFlow for deep learning
- B
scikit-learn for distributed processing and JAX for deep learning
- C
PyTorch for distributed processing and TensorFlow for deep learning
- D
Apache Spark for distributed processing and PyTorch for deep learning
- E
JAX for distributed processing and TensorFlow for deep learning
Show answer and explanation
Correct answers: A, D
Explanation
To process a large dataset with distributed computing and later train a deep learning model on GPUs, you need separate frameworks optimized for each task. Apache Spark is the ideal choice for distributed feature engineering tasks, as it is designed for large-scale data processing. For deep learning, TensorFlow and PyTorch are both suitable as they support GPU acceleration for training complex models. Thus, the correct solutions are Apache Spark with TensorFlow or Apache Spark with PyTorch.
- A. Correct.
Correct. Apache Spark is a powerful framework for distributed data processing, and TensorFlow is a widely-used framework for deep learning that supports GPU acceleration. This combination efficiently handles both distributed feature engineering and GPU-based model training.
- B. Incorrect.
Incorrect. While scikit-learn is a great library for machine learning, it is not designed for distributed processing at scale. JAX is a strong choice for deep learning but wouldn't meet the distributed requirement for feature engineering.
- C. Incorrect.
Incorrect. PyTorch is a deep learning framework, not a distributed processing framework. TensorFlow is suitable for deep learning but cannot replace the distributed processing step.
- D. Correct.
Correct. Apache Spark handles distributed processing at scale efficiently, and PyTorch is a robust choice for deep learning that supports GPU acceleration. This combination is a valid solution for the given requirements.
- E. Incorrect.
Incorrect. JAX is primarily designed for high-performance numerical computing and deep learning, but it is not a distributed processing framework. TensorFlow is a good deep learning framework but cannot fulfill the distributed processing requirement in this scenario.