DEA-C01 Question 292
Select 2You are a data engineer for a company that processes large volumes of e-commerce transaction data daily. The data is ingested into an Amazon S3 bucket, and you need to automate a workflow that performs the following tasks: 1) Validate incoming data files, 2) Transform the data into a target schema, and 3) Load the transformed data into an Amazon Redshift cluster. Which combination of AWS services should you use to automate this data processing workflow?
- A
AWS Glue for data transformation, AWS Lambda for validation, and Amazon Redshift COPY command for loading
- B
Amazon EMR for data transformation, AWS Step Functions for orchestration, and Amazon Redshift COPY command for loading
- C
AWS Data Pipeline for orchestration, AWS Glue for data transformation, and Amazon Redshift COPY command for loading
- D
Amazon Kinesis Data Streams for ingestion, AWS Glue for data transformation, and Amazon Redshift Spectrum for querying
- E
AWS Glue for validation and transformation, AWS Step Functions for orchestration, and Amazon Redshift COPY command for loading
Show answer and explanation
Correct answers: A, E
Explanation
To automate the data processing workflow, you need services that specialize in validation, transformation, orchestration, and loading. AWS Glue is a serverless tool for data transformation (and can also handle validation), AWS Lambda is suitable for lightweight validation tasks, and AWS Step Functions can orchestrate the workflow. Finally, the Amazon Redshift COPY command is optimized for loading large datasets efficiently into Redshift. Using these serverless services ensures scalability and cost-efficiency.
- A. Correct.
Correct. AWS Glue can transform data, AWS Lambda can validate incoming files, and the Amazon Redshift COPY command can efficiently load data into Redshift. This combination automates the required steps.
- B. Incorrect.
Partially correct but not the best choice. While Amazon EMR can handle data transformation and AWS Step Functions can orchestrate workflows, this setup may require more overhead compared to AWS Glue's serverless capabilities.
- C. Incorrect.
Incorrect. While AWS Data Pipeline can orchestrate workflows, it is largely considered outdated, and AWS Glue is more suited for serverless data transformation.
- D. Incorrect.
Incorrect. Amazon Kinesis Data Streams is designed for streaming data ingestion, not batch ingestion from S3. Additionally, Amazon Redshift Spectrum is used for querying external data directly from S3, not for performing bulk loads into Redshift.
- E. Correct.
Correct. AWS Glue can handle both validation and data transformation, AWS Step Functions is ideal for orchestrating the workflow, and the Amazon Redshift COPY command efficiently loads data into Redshift.