DEA-C01 Question 396
Select 3You are working on an ETL pipeline in AWS Glue to process data from an S3 bucket. During the data processing, you observe that certain partitions in your dataset take significantly longer to process due to data skew. How can you mitigate this issue using AWS Glue?
- A
Use AWS Glue's built-in DynamicFrame repartitioning to evenly distribute the data.
- B
Implement a custom hash key distribution mechanism to redistribute skewed data across partitions.
- C
Increase the number of workers in your AWS Glue job to handle the skewed data more efficiently.
- D
Leverage the 'SALTING' technique by adding randomness to the partition keys of skewed data.
- E
Enable AWS Glue Auto Scaling to dynamically adjust resources based on workload.
Show answer and explanation
Correct answers: A, B, D
Explanation
Data skew in AWS Glue can lead to uneven workloads across partitions, slowing down your ETL jobs. To mitigate this, repartitioning DynamicFrames, using custom hash keys, or applying the SALTING technique can redistribute the data more evenly. Increasing resources or enabling Auto Scaling may help with processing capacity but won't directly address the skew issue.
- A. Correct.
AWS Glue supports repartitioning of DynamicFrames, which helps in redistributing data across partitions to mitigate data skew.
- B. Correct.
Custom hash key distribution allows you to redistribute data effectively by balancing the load across partitions, reducing skew.
- C. Incorrect.
Increasing the number of workers addresses resource limitations but does not directly resolve the root cause of data skew.
- D. Correct.
SALTING adds randomness to partition keys, breaking up large skewed partitions into smaller, more manageable ones, which is an effective technique.
- E. Incorrect.
While AWS Glue Auto Scaling manages resource adjustments, it does not address the underlying issue of skewed data distribution.