Databricks Data Engineer Associate Question 49
Single answerYou are working on a Databricks cluster running a streaming job that processes data from an event hub. While monitoring the cluster, you notice that the job has stopped responding, and logs indicate that the Spark driver has run out of memory. What is the best course of action to resolve the issue?
- A
Restart the cluster to reset the Spark driver and free up memory resources.
- B
Reconfigure the cluster to use a smaller instance type and restart it.
- C
Manually terminate the Spark driver process and allow it to restart automatically.
- D
Increase the number of worker nodes in the cluster without restarting it.
Show answer and explanation
Correct answer: A
Explanation
Restarting the cluster is a common solution when the Spark driver becomes unresponsive due to memory exhaustion or other resource-related issues. It resets all cluster processes, clearing memory and allowing jobs to restart in a clean state. Other options, such as reconfiguring with smaller instances or adding worker nodes, do not address the root cause of driver memory exhaustion effectively.
- A. Correct.
Restarting the cluster will reset the Spark driver, freeing up memory resources and allowing the streaming job to resume processing. This is the most effective solution when the driver is unresponsive due to memory exhaustion.
- B. Incorrect.
Reconfiguring the cluster with a smaller instance type is unlikely to resolve the memory issue and may worsen the problem by reducing available resources.
- C. Incorrect.
Manually terminating the Spark driver process is not a recommended or feasible approach in Databricks. Restarting the entire cluster is a cleaner and more reliable solution.
- D. Incorrect.
Increasing the number of worker nodes will not address issues related to the Spark driver running out of memory, as the driver resides on the cluster's head node.