DEA-C01 Question 399
Select 2You are designing a data processing pipeline in AWS Glue. During the processing of a large dataset, you notice that certain partitions receive significantly more records than others, causing long execution times and inefficient resource utilization. Which of the following strategies can you use to mitigate data skew in your AWS Glue job?
- A
Use a custom partitioning key to evenly distribute the records across partitions.
- B
Implement a salting mechanism by appending a random value to the partition key.
- C
Increase the number of AWS Glue workers to handle the larger partitions.
- D
Use the AWS Glue DynamicFrame's resolveChoice method to handle skewed partitions.
- E
Group smaller partitions together to reduce the number of overall partitions.
Show answer and explanation
Correct answers: A, B
Explanation
Data skew occurs when certain partitions in a distributed processing environment receive significantly more records than others, leading to inefficiencies. To mitigate this in AWS Glue, you can use strategies such as implementing a custom partitioning key or salting the partition key. These methods ensure a more even data distribution across partitions. Simply increasing workers or grouping partitions does not resolve the skew itself, and features like resolveChoice are unrelated to partitioning.
- A. Correct.
Using a custom partitioning key can help distribute the records more evenly across partitions, reducing data skew.
- B. Correct.
Salting involves adding a random value to the partition key to create more even distribution across partitions, which helps mitigate skew.
- C. Incorrect.
Increasing the number of AWS Glue workers may handle larger partitions temporarily but does not address the root cause of data skew.
- D. Incorrect.
The resolveChoice method in AWS Glue is used for schema resolution, not for handling data skew.
- E. Incorrect.
Grouping smaller partitions together reduces the number of partitions but does not address the underlying data distribution issues that cause skew.