Google Professional Machine Learning Engineer Question 159
Select 3Google Cloud PlatformYou are working as a Machine Learning Engineer for a company that processes large-scale datasets to train recommendation models. The data is stored in Google Cloud Storage (GCS), and your team uses Dataproc Jupyter notebooks with Spark kernels for preprocessing. During preprocessing, you need to join multiple large datasets and apply custom transformations while ensuring scalability and efficiency. Which of the following steps would you take to achieve this?
- A
Use the Spark DataFrame API to load the datasets from GCS and perform the transformations.
- B
Increase the executor memory and core configurations in the Dataproc cluster to handle large-scale data efficiently.
- C
Write the preprocessing logic in Python pandas and execute it directly in the Spark kernel for simplicity.
- D
Persist intermediate results in memory using Spark caching mechanisms to optimize repeated operations.
- E
Disable dynamic resource allocation in the Dataproc cluster to ensure fixed resource usage for all tasks.
Show answer and explanation
Correct answers: A, B, D
Explanation
To preprocess large-scale datasets using Spark kernels in Dataproc, you should leverage distributed computing capabilities provided by Spark, such as the DataFrame API and caching mechanisms, to efficiently handle data transformations. Adjusting executor configurations ensures the cluster has sufficient resources for large operations. Avoid using non-distributed frameworks like pandas in this scenario, and enable dynamic resource allocation to optimize cluster performance.
- A. Correct.
Correct: The Spark DataFrame API is optimized for distributed data processing and is a scalable approach for handling large datasets in a Spark kernel. It integrates well with GCS for data loading and transformations.
- B. Correct.
Correct: Increasing executor memory and core configurations can help the Dataproc cluster handle large-scale data processing tasks more effectively, especially when dealing with resource-intensive operations.
- C. Incorrect.
Incorrect: While pandas is great for small-scale data processing, it is not optimized for distributed data processing like Spark. Using pandas in a Spark kernel negates Spark's scalability benefits.
- D. Correct.
Correct: Persisting intermediate results in memory using Spark caching mechanisms can improve performance by avoiding redundant computations during iterative operations.
- E. Incorrect.
Incorrect: Disabling dynamic resource allocation is not recommended as it prevents the cluster from adapting to varying workload demands, potentially leading to inefficient resource usage.