Databricks Machine Learning Associate Question 338
Single answerA data engineering team currently processes a large dataset using Pandas in a single-node environment. Due to increasing data volume, they are facing memory issues and slow execution times. The team wants to scale their pipeline without significant code refactoring. Which solution would be most appropriate?
- A
Use Pandas API on Spark to distribute computations across a cluster.
- B
Switch to Apache Hive for distributed data processing.
- C
Rewrite the entire pipeline using PySpark DataFrame API.
- D
Increase the memory of the single-node machine to handle larger datasets.
Show answer and explanation
Correct answer: A
Explanation
The Pandas API on Spark is designed to provide a scalable solution for users familiar with Pandas who need to process large datasets in a distributed environment. By using this API, the team can scale their pipeline with minimal code changes, unlike other options that require significant refactoring or do not address scalability effectively.
- A. Correct.
Correct: Pandas API on Spark allows users to scale their Pandas workloads across a Spark cluster while requiring minimal code changes, making it an ideal solution for this scenario.
- B. Incorrect.
Incorrect: Apache Hive is primarily used for querying and analyzing structured data. It would require significant changes to the current pipeline and is not designed for Pandas-like workflows.
- C. Incorrect.
Incorrect: While PySpark DataFrame API is a powerful option, rewriting the entire pipeline would involve significant refactoring, which the team wants to avoid.
- D. Incorrect.
Incorrect: Increasing memory on a single machine might temporarily alleviate the issue, but it does not address scalability as the dataset continues to grow.