Google Professional Machine Learning Engineer Question 161
Single answerGoogle Cloud PlatformYou are a Machine Learning Engineer tasked with preprocessing a large dataset containing billions of log records stored in Google Cloud Storage (GCS). The preprocessing involves complex transformations, including aggregations and filtering. You want to use a Jupyter Notebook to prototype and scale your workload efficiently. How can you achieve this using Spark kernels in Google Cloud?
- A
Use a Dataproc cluster with JupyterLab and a Spark kernel, and connect it to your GCS data for distributed processing.
- B
Run a Jupyter Notebook on your local machine and install the PySpark library for Spark kernel support.
- C
Use BigQuery instead of Spark kernels because Spark cannot handle distributed processing on large datasets.
- D
Deploy a Vertex AI Workbench Managed Notebook with a pre-installed Spark kernel and directly process the data in GCS.
Show answer and explanation
Correct answer: A
Explanation
Dataproc provides a scalable and managed environment for running Apache Spark workloads. By enabling JupyterLab with Spark kernels on a Dataproc cluster, you can efficiently prototype and execute distributed data preprocessing tasks on large datasets stored in Google Cloud Storage. This approach is specifically designed to handle large-scale transformations and aggregations, making it the best choice for this scenario.
- A. Correct.
Correct: Dataproc provides managed clusters for Spark, and JupyterLab with a Spark kernel allows you to prototype and execute distributed preprocessing on large datasets stored in GCS.
- B. Incorrect.
Incorrect: Running on a local machine with PySpark might work for small-scale testing but will not scale efficiently for distributed processing of billions of records.
- C. Incorrect.
Incorrect: While BigQuery is a powerful tool for large datasets, this question specifically asks about using Spark kernels, which are suitable for distributed processing and transformations.
- D. Incorrect.
Incorrect: Vertex AI Workbench Managed Notebooks do not natively support Spark kernels for distributed processing. Dataproc is the appropriate choice for this use case.