Databricks Machine Learning Associate Question 215
Select 3A data scientist is training a large machine learning model on a Databricks cluster with 16 worker nodes using distributed training. During training, they observe that increasing the number of worker nodes to 32 significantly increases the job runtime rather than reducing it. Which of the following could be contributing to this issue?
- A
The dataset is too small relative to the number of worker nodes, causing overhead in parallelization.
- B
The model's training process involves frequent synchronization steps, leading to bottlenecks as the number of nodes increases.
- C
The compute resources per worker node are insufficient for the model's requirements, causing resource contention.
- D
The cluster's driver node is under-provisioned, leading to slower task scheduling and coordination.
- E
Increasing the number of worker nodes always improves performance, so the issue lies elsewhere.
Show answer and explanation
Correct answers: A, B, D
Explanation
In distributed systems, increasing the number of worker nodes can lead to diminishing returns or even degraded performance if the overhead of distributing tasks and synchronizing results outweighs the computational benefits. Small datasets, synchronization-heavy algorithms, and an under-provisioned driver node are common causes of performance degradation when scaling out in a Databricks environment.
- A. Correct.
If the dataset is too small relative to the number of worker nodes, the overhead of managing parallelization (e.g., task distribution and inter-node communication) can outweigh the benefits of parallel processing.
- B. Correct.
Some machine learning algorithms require frequent synchronization across nodes (e.g., gradient aggregation in distributed training). This can create bottlenecks as the number of nodes increases, reducing performance.
- C. Incorrect.
Insufficient compute resources per worker node could cause issues, but in this scenario, the problem arises specifically from the increase in worker nodes, not resource contention on individual nodes.
- D. Correct.
An under-provisioned driver node can slow down task scheduling and coordination, particularly in a larger cluster, leading to increased runtime.
- E. Incorrect.
This statement is incorrect because increasing the number of worker nodes does not always improve performance. Factors like synchronization overhead and resource allocation must be considered.