DEA-C01 Question 78
Single answerYou are designing a data pipeline to process sales data stored in Amazon S3 and load it into an Amazon Redshift cluster for analytics. The pipeline must include data transformation and error handling. You want to orchestrate this pipeline using AWS services in a cost-effective and serverless manner. Which service or combination of services is the most appropriate for orchestrating this pipeline?
- A
AWS Step Functions to orchestrate Lambda functions for data transformation and loading
- B
AWS Glue Workflows to manage and orchestrate the ETL process
- C
Amazon EC2 instances with custom scripts for orchestration
- D
AWS Data Pipeline for end-to-end orchestration
Show answer and explanation
Correct answer: B
Explanation
AWS Glue Workflows is the most appropriate service for orchestrating this pipeline because it is specifically designed for managing ETL processes in a serverless and cost-effective manner. It integrates seamlessly with other AWS services like S3 and Redshift, and it provides built-in capabilities for error handling and data transformation. While other options like Step Functions or Data Pipeline could potentially be used, they are either less suited for ETL tasks or no longer the best practice for modern data pipelines.
- A. Incorrect.
AWS Step Functions can orchestrate Lambda functions, but for large-scale data transformation and loading tasks, AWS Glue Workflows is a better choice due to its built-in ETL capabilities.
- B. Correct.
AWS Glue Workflows is specifically designed for orchestrating ETL pipelines, making it a cost-effective and serverless solution for this use case.
- C. Incorrect.
Using Amazon EC2 instances with custom scripts is not cost-effective and requires additional management overhead compared to serverless options like AWS Glue Workflows.
- D. Incorrect.
AWS Data Pipeline is a legacy service and is less suitable compared to AWS Glue Workflows for modern serverless ETL orchestration.