Databricks Machine Learning Associate Question 275
Single answerA data scientist is working on a machine learning project using Databricks. The dataset being used contains 500,000 rows and fits comfortably into the memory of a single node. The data scientist needs to quickly train and evaluate a regression model. Given these requirements, why might they choose scikit-learn over Spark ML?
- A
scikit-learn is optimized for distributed computing, making it faster for large-scale datasets.
- B
scikit-learn is a single-node library, making it well-suited for smaller datasets that fit into memory.
- C
Spark ML can only process distributed datasets, making it unsuitable for single-node computations.
- D
scikit-learn provides more prebuilt machine learning algorithms compared to Spark ML.
Show answer and explanation
Correct answer: B
Explanation
scikit-learn is a single-node solution, making it better suited for datasets that fit entirely into memory and don't require distributed computing. In this scenario, the dataset is small enough to fit into memory, so using scikit-learn would likely be faster and more efficient compared to Spark ML, which is optimized for distributed workloads.
- A. Incorrect.
This is incorrect because scikit-learn is not optimized for distributed computing. Instead, it is designed for single-node computations.
- B. Correct.
This is correct because scikit-learn is a single-node solution, which makes it ideal for smaller datasets that fit entirely into memory.
- C. Incorrect.
This is incorrect because Spark ML can also handle smaller datasets, though it is primarily designed for distributed computations, making it less efficient in single-node scenarios.
- D. Incorrect.
This is partially true but irrelevant to the scenario. While scikit-learn does offer a wide range of machine learning algorithms, the question focuses on its suitability for single-node computations.