Databricks Data Engineer Professional Question 52
Single answerA data engineer is working on optimizing a large dataset for downstream processing in Databricks. The dataset is currently partitioned into 1,000 partitions, which is causing performance bottlenecks during shuffle operations. The engineer wants to reduce the number of partitions to 100 without changing the data distribution, but they also want to minimize the computational cost of this operation. Which method should the engineer use?
- A
coalesce
- B
repartition
- C
repartitionByRange
- D
rebalance
Show answer and explanation
Correct answer: A
Explanation
When reducing the number of partitions in a dataset, coalesce is the most efficient method as it avoids a full shuffle by merging adjacent partitions. This makes it ideal for scenarios where the goal is to decrease partitions without redistributing data. Other methods like repartition would incur higher computational costs due to shuffling, and repartitionByRange or rebalance are not applicable to the problem described in this scenario.
- A. Correct.
coalesce is the correct choice for reducing the number of partitions without causing a full shuffle, thus minimizing computational cost.
- B. Incorrect.
repartition performs a full shuffle when changing the number of partitions, which is computationally more expensive than coalesce and not necessary in this scenario.
- C. Incorrect.
repartitionByRange is used to repartition data based on a specified column or range, which is not required here as the goal is only to reduce the number of partitions.
- D. Incorrect.
rebalance is used to evenly distribute partitions across cluster nodes, but it is not suited for simply reducing the number of partitions.