DEA-C01 Question 80
Single answerYou are working as a data engineer for a company that processes large volumes of data daily. You need to design and orchestrate a data pipeline that performs the following tasks: fetch data from an S3 bucket, transform it using a Python script, and load the transformed data into an Amazon Redshift cluster. The pipeline must run on a daily schedule and provide retry mechanisms in case of failures. Which AWS service should you use to orchestrate this pipeline?
- A
AWS Glue Workflows
- B
AWS Step Functions
- C
AWS Data Pipeline
- D
Amazon MWAA (Managed Workflows for Apache Airflow)
Show answer and explanation
Correct answer: A
Explanation
AWS Glue Workflows is specifically designed to orchestrate data pipelines involving data extraction, transformation, and loading (ETL) tasks. It integrates natively with services like S3 and Redshift, supports Python scripts via Glue jobs, and provides scheduling and retry mechanisms, making it the most efficient and straightforward choice for this scenario.
- A. Correct.
AWS Glue Workflows is the most suitable choice because it is designed for orchestrating data pipelines and integrates seamlessly with S3, Redshift, and Python scripts. It also supports scheduling and retry mechanisms.
- B. Incorrect.
AWS Step Functions is a general-purpose orchestration service, but it is not specifically optimized for data-related workflows like AWS Glue Workflows. It would require more setup to handle this specific use case.
- C. Incorrect.
AWS Data Pipeline is a deprecated service and is not recommended for new workflows. It does not provide the same level of integration with modern AWS services like Glue or Redshift.
- D. Incorrect.
Amazon MWAA (Managed Workflows for Apache Airflow) is a valid option for orchestrating complex workflows, but it would require more setup and customization compared to AWS Glue Workflows. It is better suited for cases where customers are already using Apache Airflow.