Databricks Data Engineer Professional Question 54
Single answerYou are working on a large dataset in Databricks and notice that some partitions are highly skewed, causing performance issues during downstream transformations. You also need to ensure that the partitioning respects a specific column's value range. Which partitioning method should you use in this scenario?
- A
coalesce
- B
repartition
- C
repartition by range
- D
rebalance
Show answer and explanation
Correct answer: C
Explanation
When dealing with skewed partitions and the need to respect a specific column's value range, 'repartition by range' is the most appropriate method. It distributes the data into partitions based on the range of values in the specified column, ensuring that the partitions are more balanced and optimized for downstream operations. Other methods, such as 'coalesce' and 'rebalance,' either do not address the skew properly or do not respect column value ranges.
- A. Incorrect.
Coalesce is used to reduce the number of partitions in a dataset, but it does not allow for control over the partitioning based on column values. It is more appropriate for reducing partition overhead at the end of a transformation pipeline.
- B. Incorrect.
Repartition increases or decreases the number of partitions and redistributes data, but it does so randomly or round-robin and does not respect column value ranges.
- C. Correct.
Repartition by range redistributes the data into partitions based on ranges of the specified column values. This is ideal for handling skewed data while ensuring that partitioning respects a specific column's value range.
- D. Incorrect.
Rebalance is used to evenly distribute data across partitions to address skew but does not consider column value ranges. It is more suitable for uniform redistribution without considering specific column values.