Databricks Machine Learning Associate Question 216
Single answerYou are working on a Databricks cluster to train a machine learning model on a large dataset. The current cluster has 4 worker nodes, each with 8 cores. However, during training, you notice that the job is taking longer than expected. Upon investigation, you find that the training job is not fully utilizing the available resources. Considering the balance between compute resources and parallelization, which action would be the most appropriate to optimize the training process?
- A
Increase the number of worker nodes in the cluster to 8 while keeping the same number of cores per node.
- B
Reduce the number of parallel tasks to match the number of cores available across the cluster.
- C
Increase the number of parallel tasks to exceed the total number of cores in the cluster.
- D
Modify the training logic to ensure that each parallel task processes a smaller batch of data.
Show answer and explanation
Correct answer: B
Explanation
The key to optimizing the training process is ensuring that the number of parallel tasks aligns with the available compute resources. When the number of tasks matches the total cores across the cluster, it avoids underutilization of resources while preventing resource contention. This balance is critical for efficient distributed computing in a Databricks environment.
- A. Incorrect.
Increasing the number of worker nodes without addressing the imbalance between parallel tasks and available cores may still lead to inefficient resource utilization. This does not directly address the issue of underutilized compute resources during training.
- B. Correct.
Reducing the number of parallel tasks to match the total number of cores ensures that each core is effectively utilized without overloading the system. This can improve the overall training performance.
- C. Incorrect.
Increasing parallel tasks beyond the available cores can lead to resource contention and degraded performance, as tasks will compete for limited compute resources.
- D. Incorrect.
While modifying the training logic to process smaller batches might help in certain scenarios, it does not directly address the issue of underutilized compute resources caused by an imbalance between parallel tasks and cores.