Databricks Machine Learning Associate Question 385
Select 3You are training a linear regression model using Spark MLlib on a large dataset with billions of rows distributed across a cluster. How does Spark scale linear regression to handle such large datasets?
- A
By using distributed gradient descent algorithms to parallelize computations across partitions.
- B
By storing all intermediate computations in memory on a single node to speed up processing.
- C
By leveraging optimized matrix factorizations that can operate on distributed data.
- D
By partitioning the data and processing each partition independently to compute local gradients.
- E
By requiring all data to fit into memory on a single executor to ensure model convergence.
Show answer and explanation
Correct answers: A, C, D
Explanation
To scale linear regression, Spark MLlib distributes computations across a cluster by using distributed gradient descent algorithms and optimized matrix operations. Data is partitioned, and computations are performed locally on each partition before aggregating results across the cluster. This approach allows Spark to handle massive datasets efficiently without requiring all the data to fit into memory on a single machine.
- A. Correct.
Correct. Spark uses distributed gradient descent algorithms, such as mini-batch gradient descent, where computations are parallelized across multiple partitions in the cluster.
- B. Incorrect.
Incorrect. Storing all intermediate computations in memory on a single node would result in bottlenecks and scalability issues. Spark distributes computations across nodes to avoid this.
- C. Correct.
Correct. Spark leverages optimized matrix operations and distributed frameworks to perform linear algebra operations efficiently on large, distributed datasets.
- D. Correct.
Correct. Spark partitions the data and computes local gradients for each partition, which are then aggregated across the cluster to update the model parameters.
- E. Incorrect.
Incorrect. Spark does not require all data to fit into memory on a single executor. Instead, it processes data in a distributed manner and can handle datasets much larger than the memory of a single machine.