DEA-C01 Question 69
Select 2You are working as a data engineer at a company that processes large volumes of customer transaction data daily. The data is stored in Amazon S3, and you are tasked with building a scalable solution to clean and transform this data before loading it into Amazon Redshift for analytics. The cleaning process involves filtering invalid transactions, removing duplicates, and performing aggregations. Which approach should you use with Apache Spark to achieve this efficiently?
- A
Use AWS Glue to run Apache Spark jobs that read the data from Amazon S3, process it, and write the cleaned data back to Amazon S3 in a format optimized for Redshift.
- B
Use an Amazon EMR cluster with Apache Spark installed to process the data, then write the cleaned data directly to Amazon Redshift using the Spark JDBC connector.
- C
Use Apache Spark on an Amazon EC2 instance to process the data and write the cleaned data to Amazon DynamoDB for further analytics.
- D
Use Apache Spark on Amazon EMR to process the data in-memory, then write the cleaned data back to Amazon S3 in Parquet format before using Redshift Spectrum to query the data.
- E
Use Apache Spark on AWS Lambda to process the data and write the cleaned data to Amazon Redshift in batches.
Show answer and explanation
Correct answers: A, B
Explanation
For processing large volumes of data with Apache Spark, AWS Glue and Amazon EMR are the most appropriate solutions within the AWS ecosystem. AWS Glue provides a serverless environment for running Spark jobs, while Amazon EMR offers a managed service for running Spark at scale on a cluster. Both solutions are capable of cleaning, transforming, and loading the data into Amazon Redshift for analytics. Using DynamoDB or Lambda in this scenario is not suitable due to their limitations, and Redshift Spectrum does not fulfill the requirement of loading data into Redshift.
- A. Correct.
This is a valid option. AWS Glue provides a serverless environment to run Apache Spark jobs for ETL tasks. It is particularly suited for processing data stored in Amazon S3 and preparing it for analytics, including optimizing the format for Redshift.
- B. Correct.
This is a valid option. Amazon EMR provides a managed environment for running Apache Spark at scale. Using the Spark JDBC connector allows for direct data insertion into Amazon Redshift after processing.
- C. Incorrect.
This is not a valid option. While Apache Spark can run on Amazon EC2, it is not the recommended approach for scalable data processing in AWS. Additionally, Amazon DynamoDB is not typically used for analytics workloads.
- D. Incorrect.
This is not ideal for this scenario. While Apache Spark on Amazon EMR can process data in-memory and Redshift Spectrum can query S3 data, this approach does not load data into Redshift itself, which is a requirement of the scenario.
- E. Incorrect.
This is not a valid option. Apache Spark is not supported on AWS Lambda, and AWS Lambda is not suitable for processing large volumes of data due to its runtime limits and stateless nature.