DEA-C01 Question 119
Select 3You are designing a CI/CD pipeline for deploying data pipelines in AWS. Your team wants to automate the testing and deployment process to minimize manual effort and ensure consistency. The data pipelines will be built using AWS Glue jobs, and the infrastructure is defined using AWS CloudFormation. Which combination of actions would best achieve these requirements?
- A
Use AWS CodePipeline to orchestrate the CI/CD process, including source control, build, test, and deployment stages.
- B
Use AWS CodeDeploy to directly deploy AWS Glue jobs into production without any testing stages.
- C
Incorporate automated unit and integration tests for AWS Glue jobs in the CI/CD pipeline using AWS CodeBuild.
- D
Define infrastructure resources such as AWS Glue jobs in a version-controlled AWS CloudFormation template.
- E
Manually test the AWS Glue jobs after deployment to ensure they work as expected.
Show answer and explanation
Correct answers: A, C, D
Explanation
To implement a CI/CD pipeline for AWS Glue-based data pipelines, it is necessary to automate the process of orchestration, testing, and deployment. AWS CodePipeline is ideal for managing the pipeline stages. AWS CodeBuild supports automated tests, which are crucial for validating the functionality of AWS Glue jobs. Using AWS CloudFormation templates for infrastructure as code ensures consistency and minimizes manual errors. Together, these approaches align with CI/CD best practices for data engineering workflows.
- A. Correct.
Correct. AWS CodePipeline is a managed service that allows you to automate the build, test, and deployment stages for CI/CD. It is a suitable choice for orchestrating the CI/CD process for data pipelines.
- B. Incorrect.
Incorrect. AWS CodeDeploy is not typically used for deploying AWS Glue jobs. Additionally, skipping testing stages goes against the principles of CI/CD.
- C. Correct.
Correct. AWS CodeBuild can be used to run automated tests, ensuring that your AWS Glue jobs are properly validated before deployment.
- D. Correct.
Correct. Defining infrastructure resources in version-controlled AWS CloudFormation templates ensures that infrastructure is consistent and repeatable, aligning with CI/CD best practices.
- E. Incorrect.
Incorrect. Manually testing after deployment introduces inefficiency and is not aligned with the goals of CI/CD, which emphasize automation.