DEA-C01 Question 358
Select 2Your team is building a data processing workflow to analyze customer transaction data stored in an Amazon S3 bucket. The data must be pre-processed daily and then loaded into an Amazon Redshift cluster for complex analytics. You want to minimize operational overhead while ensuring scalability and cost-effectiveness for the workflow. Which combination of services should you use to design this solution?
- A
Use AWS Glue for data pre-processing and AWS Glue ETL jobs to load data into Amazon Redshift.
- B
Use Amazon EMR with Apache Spark for data pre-processing and Amazon Redshift COPY command to load data.
- C
Use AWS Lambda for data pre-processing and Amazon Redshift Data API for loading data.
- D
Use Amazon Kinesis Data Firehose to process and load streaming data directly into Amazon Redshift.
- E
Use AWS Glue DataBrew for pre-processing and Amazon Redshift Spectrum for querying data directly from S3.
Show answer and explanation
Correct answers: A, B
Explanation
To design a scalable, cost-effective, and low-overhead solution for daily batch processing and loading into Amazon Redshift, AWS Glue or Amazon EMR are the best options. AWS Glue provides serverless ETL capabilities, while Amazon EMR offers flexibility and scalability for large-scale data processing. Both integrate well with Amazon Redshift for efficient data loading.
- A. Correct.
AWS Glue is a serverless data integration service that simplifies pre-processing and loading data into Amazon Redshift. It minimizes operational overhead and supports scalability.
- B. Correct.
Amazon EMR with Apache Spark is suitable for large-scale data processing, and using the Amazon Redshift COPY command can efficiently load processed data into Redshift.
- C. Incorrect.
AWS Lambda is not suitable for large-scale ETL workflows due to its execution time and memory limits. It is better suited for lightweight, event-driven tasks.
- D. Incorrect.
Amazon Kinesis Data Firehose is designed for streaming data ingestion, but the scenario involves daily batch processing, which makes it unsuitable.
- E. Incorrect.
AWS Glue DataBrew is for no-code data preparation, but it is not meant for production-ready ETL pipelines. Amazon Redshift Spectrum is used for querying data directly from S3, not for pre-processing and loading into Redshift.