Databricks Machine Learning Associate Question 342
Single answerYou are working with a large dataset that exceeds the memory capacity of your local machine. Your current data processing code is written using Pandas, but you need to scale it to handle this large dataset without rewriting most of the logic. Which approach would be the most appropriate to achieve this in Databricks?
- A
Use the Pandas API on Spark to scale your existing Pandas code with minimal refactoring.
- B
Rewrite the entire pipeline using PySpark DataFrame API for better scalability.
- C
Use Dask to distribute the workload across multiple nodes in the Databricks cluster.
- D
Optimize the existing Pandas code by increasing the memory limit on your local machine.
Show answer and explanation
Correct answer: A
Explanation
The Pandas API on Spark allows you to scale your existing Pandas workflows to large datasets by leveraging the distributed computing power of Spark. This solution requires minimal changes to your existing code, making it ideal for scenarios where you need scalability without extensive refactoring.
- A. Correct.
This is correct because the Pandas API on Spark provides a scalable way to run Pandas-like operations on large datasets with minimal code changes.
- B. Incorrect.
This is incorrect because rewriting the pipeline using PySpark DataFrame API would require significant refactoring, which the question explicitly states should be avoided.
- C. Incorrect.
This is incorrect because while Dask is a distributed computing framework, it is not natively integrated with Databricks and would require additional setup effort.
- D. Incorrect.
This is incorrect because increasing memory on the local machine does not solve the problem of scaling to large datasets that exceed local resources.