Databricks Data Engineer Professional Question 240
Select 3You are troubleshooting a Spark job that is taking significantly longer than expected to complete. Upon reviewing the Spark UI, you observe a high task serialization time and frequent garbage collection pauses. In the Ganglia UI, the memory usage on the worker nodes is consistently near the maximum limit. What actions should you take to improve the job's performance?
- A
Increase the executor memory to reduce garbage collection overhead.
- B
Increase the number of partitions to balance the workload more evenly across tasks.
- C
Set the 'spark.serializer' configuration to KryoSerializer to reduce task serialization time.
- D
Reduce the number of shuffle partitions to decrease shuffle data size.
- E
Decrease the number of executors to allow each executor to use more memory.
Show answer and explanation
Correct answers: A, B, C
Explanation
The observations from the Spark UI and Ganglia UI point to two main issues: high task serialization time and frequent garbage collection due to memory pressure. Increasing executor memory addresses high garbage collection time, while increasing partitions helps balance the workload and reduce memory contention. Switching to KryoSerializer can optimize serialization performance, addressing the high task serialization time. The other options would either worsen the problem or fail to address the root causes.
- A. Correct.
Increasing executor memory can reduce garbage collection overhead by allowing more space for objects in memory, thereby reducing frequent collections.
- B. Correct.
Increasing the number of partitions can help balance the workload across tasks, leading to better resource utilization and performance.
- C. Correct.
Using KryoSerializer can reduce the task serialization time as it is faster and more efficient than the default Java serializer.
- D. Incorrect.
Reducing the number of shuffle partitions can lead to larger partitions, which may increase memory pressure and worsen garbage collection issues, contrary to what is needed in this scenario.
- E. Incorrect.
Decreasing the number of executors reduces parallelism and can worsen the performance issue by overloading the remaining executors, especially since memory is already constrained.