Databricks Data Engineer Professional Question 53
Single answerYou are working on optimizing a large dataset in a Databricks pipeline. The dataset contains sales records for multiple regions and is heavily skewed towards certain regions. You need to redistribute the data evenly across partitions to improve parallelism for subsequent transformations. Which partitioning technique would be the most appropriate to achieve this?
- A
coalesce
- B
repartition
- C
repartitionByRange
- D
rebalance
Show answer and explanation
Correct answer: D
Explanation
When dealing with skewed data in a Databricks pipeline, the 'rebalance' method is specifically designed to redistribute data evenly across partitions. This ensures better parallelism and avoids partition hotspots. While other methods like 'repartition' or 'repartitionByRange' can redistribute data, they are not optimal for explicitly addressing data skew.
- A. Incorrect.
coalesce reduces the number of partitions but does not redistribute data evenly across partitions. It is not suitable for addressing skewed data.
- B. Incorrect.
repartition increases or decreases the number of partitions and performs a full shuffle. However, it does not explicitly handle data skew caused by uneven distributions.
- C. Incorrect.
repartitionByRange redistributes data based on ranges of column values and is useful for range-based queries but does not guarantee even distribution to address skew.
- D. Correct.
rebalance explicitly redistributes data evenly across partitions, making it the most appropriate technique for addressing skewed data.