DEA-C01 Question 117
Select 3You are a data engineer building an end-to-end data pipeline using AWS services. The pipeline ingests streaming data from Amazon Kinesis Data Streams, processes it using AWS Glue, and stores the results in Amazon S3. Your team wants to implement a CI/CD process to automate testing and deployment of the pipeline. Which combination of AWS services and features should you use to achieve this?
- A
AWS CodePipeline to orchestrate the CI/CD process
- B
AWS CodeBuild to run unit tests on the AWS Glue scripts
- C
AWS CloudFormation to define the infrastructure as code for the pipeline
- D
Amazon QuickSight to monitor the status of the CI/CD pipeline
- E
AWS CodeDeploy to deploy the AWS Glue jobs
Show answer and explanation
Correct answers: A, B, C
Explanation
To automate the CI/CD process for a data pipeline, you need a combination of tools that handle orchestration (AWS CodePipeline), testing (AWS CodeBuild), and infrastructure management (AWS CloudFormation). These services work together to ensure that the pipeline is properly tested, deployed, and maintained in a repeatable and automated manner. Amazon QuickSight and AWS CodeDeploy are not suitable for these specific tasks.
- A. Correct.
Correct. AWS CodePipeline is used to orchestrate CI/CD processes by automating the steps involved in building, testing, and deploying applications or data pipelines.
- B. Correct.
Correct. AWS CodeBuild can be used to run unit tests and validate the AWS Glue scripts during the CI/CD process.
- C. Correct.
Correct. AWS CloudFormation allows you to define and provision the infrastructure required for the data pipeline in a repeatable and automated way, which is a critical component of CI/CD.
- D. Incorrect.
Incorrect. While Amazon QuickSight is useful for data visualization and reporting, it is not designed to monitor or manage CI/CD pipelines.
- E. Incorrect.
Incorrect. AWS CodeDeploy is used for application deployment, such as deploying code to EC2 or Lambda, but it is not suitable for deploying AWS Glue jobs.