DEA-C01 Question 397
Select 2You are designing an Amazon EMR-based solution to process a large dataset stored in Amazon S3 using Apache Spark. During testing, you notice that one of the tasks is significantly slower than others, due to data skew caused by a single key being associated with a disproportionately large number of records. Which of the following approaches can help mitigate this issue?
- A
Add a random salt value to the skewed key and split the data into smaller partitions.
- B
Leverage the Spark
coalesce()function to reduce the number of partitions. - C
Use the
replicate skewed datatechnique to duplicate skewed keys across multiple partitions. - D
Filter out the skewed key from the dataset to avoid processing it.
- E
Use Spark's
groupByKey()operation to process the data in memory.
Show answer and explanation
Correct answers: A, C
Explanation
Data skew occurs when one or more keys in a dataset have a disproportionately large number of associated records, causing performance bottlenecks. To address this, adding a random salt to the skewed key helps distribute the records across multiple partitions. Similarly, replicating the skewed data allows the processing workload to be distributed more effectively. Techniques like reducing partitions (coalesce()), removing data, or using inefficient operations (groupByKey()) are not suitable for mitigating data skew in distributed data processing frameworks like Apache Spark.
- A. Correct.
Adding a random salt value to the skewed key helps distribute the records associated with the skewed key into multiple partitions, mitigating the data skew issue.
- B. Incorrect.
Using the
coalesce()function reduces the number of partitions, which can further aggravate data skew by consolidating data into fewer partitions, making the issue worse. - C. Correct.
Replicating skewed data allows the workload to be distributed across multiple partitions, ensuring that the processing of the skewed key does not bottleneck a single task.
- D. Incorrect.
Filtering out the skewed key completely is not a valid solution, as it would result in data loss and compromise the completeness of the dataset.
- E. Incorrect.
Using
groupByKey()is inefficient for large datasets and can exacerbate memory issues, especially when dealing with skewed data.