DEA-C01 Question 393
Single answerYou are working on a data pipeline in AWS Glue that processes a large dataset stored in Amazon S3. To optimize the pipeline's performance, you decide to sample the data for testing purposes. Which of the following sampling techniques in AWS Glue would allow you to randomly select a subset of the data while maintaining a representative distribution?
- A
Splitting the dataset into equal-sized partitions and selecting one partition
- B
Using AWS Glue's built-in 'RandomSplit' transformation
- C
Filtering the dataset by a specific condition (e.g., selecting rows where a column value > 100)
- D
Using AWS Glue's built-in 'Top Rows' transformation to select the first N rows
Show answer and explanation
Correct answer: B
Explanation
AWS Glue's 'RandomSplit' transformation is specifically designed for random sampling, allowing you to split the dataset into subsets with representative distributions. Other techniques, such as selecting partitions, applying filters, or choosing the first N rows, do not ensure randomness and may lead to biased results.
- A. Incorrect.
Splitting the dataset into equal-sized partitions and selecting one partition may not guarantee a representative distribution as it depends on the data's natural order.
- B. Correct.
Using AWS Glue's built-in 'RandomSplit' transformation is the correct approach for randomly selecting a representative subset of the data.
- C. Incorrect.
Filtering the dataset by a specific condition, such as selecting rows based on column values, introduces bias and is not a random sampling technique.
- D. Incorrect.
Using the 'Top Rows' transformation to select the first N rows is not random and could result in a biased or unrepresentative sample if the dataset is sorted.