Databricks Data Engineer Professional Question 238
Select 2You are debugging a Spark job that is running slowly on a Databricks cluster. Upon reviewing the Spark UI, you notice that the 'Tasks' page shows a high ratio of tasks in the 'Pending' state, while the 'Stages' page indicates a skew in task durations across partitions. Additionally, the Ganglia UI reveals that the cluster's CPU usage is uneven across worker nodes. What are the most likely causes of the performance issue?
- A
Data skew causing certain tasks to process disproportionately large partitions
- B
Worker nodes being under-provisioned in terms of memory
- C
A bottleneck in the cluster's network bandwidth
- D
A poorly tuned Spark configuration, such as an incorrectly set shuffle partition count
- E
Excessive garbage collection on the driver node due to large broadcast variables
Show answer and explanation
Correct answers: A, D
Explanation
The most likely causes of the described performance issues are data skew and a poorly tuned Spark configuration, particularly involving shuffle partitions. Data skew explains the disparity in task durations, while an incorrect shuffle partition count could lead to uneven CPU usage across nodes. The other options, while plausible in other scenarios, do not align with the specific symptoms observed in the Spark UI and Ganglia UI.
- A. Correct.
Data skew can cause some tasks to take significantly longer than others because certain partitions contain more data. This aligns with the task duration skew observed in the Spark UI.
- B. Incorrect.
While under-provisioned memory can cause performance issues, the symptoms described (task skew and uneven CPU usage) do not directly point to a memory issue.
- C. Incorrect.
Network bandwidth bottlenecks often manifest as slow data shuffles or high network latencies, but the Spark UI and Ganglia UI symptoms described do not specifically suggest this issue.
- D. Correct.
An incorrectly set shuffle partition count can lead to suboptimal parallelism, which may result in uneven task distribution and performance issues. This matches the observations of task skew and uneven CPU usage.
- E. Incorrect.
Excessive garbage collection on the driver node would manifest as delays in job scheduling and driver activity, but the issue described pertains to task execution and partition skew, not driver memory management.