Databricks Machine Learning Associate Question 341
Single answerYou are working on a machine learning pipeline where the data preprocessing steps are written using Pandas. Your dataset has grown significantly, and now you are encountering memory and performance issues while running the pipeline. You want to scale your data processing without rewriting the existing codebase. Which solution should you choose?
- A
Rewrite the entire pipeline using Apache Spark DataFrame APIs
- B
Switch to Pandas API on Spark to scale the pipeline with minimal code changes
- C
Use Dask to distribute your existing Pandas code
- D
Upgrade the hardware to handle larger datasets
Show answer and explanation
Correct answer: B
Explanation
Pandas API on Spark is specifically designed to scale Pandas workflows using the distributed computing power of Apache Spark while requiring minimal code refactoring. This makes it the most efficient and practical solution for scaling data pipelines in Databricks when the goal is to avoid rewriting large portions of code.
- A. Incorrect.
While rewriting the pipeline with Apache Spark DataFrame APIs would scale it, this approach requires significant refactoring, contradicting the goal of minimal code changes.
- B. Correct.
Switching to Pandas API on Spark allows you to scale your existing Pandas code with minimal changes by leveraging the distributed computing power of Spark, making it the ideal choice in this scenario.
- C. Incorrect.
Dask is another solution for scaling Pandas workflows, but it requires installing and setting up a new library, and it is not directly related to leveraging Databricks' native capabilities.
- D. Incorrect.
Upgrading hardware might temporarily address memory issues, but it is not a scalable or cost-effective solution for handling large datasets in a distributed manner.