DEA-C01 Question 304
Select 3A data engineering team processes customer transaction data daily using an AWS Glue ETL job and stores the results in Amazon S3. Recently, the team noticed unexpected discrepancies in the output, leading to incorrect business insights. You are tasked with ensuring the data processing pipeline consistently produces accurate and repeatable results. What actions should you take to address the issue?
- A
Enable AWS Glue job bookmarking to ensure incremental processing of new data.
- B
Implement input data validation checks before processing to ensure data quality.
- C
Use AWS Glue job parameters to pass dynamic input configurations during ETL execution.
- D
Enable CloudWatch Logs for the AWS Glue job to monitor errors and troubleshoot failures.
- E
Switch to an on-demand AWS Lambda function for ETL processing for better scalability.
Show answer and explanation
Correct answers: A, B, D
Explanation
To ensure consistent and repeatable business outcomes, you must address potential data quality issues, monitor processing for errors, and optimize the ETL pipeline for incremental data processing. AWS Glue job bookmarking prevents reprocessing old data, input data validation ensures only clean data is processed, and CloudWatch Logs provide insights into errors for troubleshooting. Together, these actions help maintain and troubleshoot the pipeline effectively.
- A. Correct.
Enabling AWS Glue job bookmarking ensures that the ETL job processes only new or updated data, which helps maintain consistency and avoids reprocessing errors. This is critical for repeatable business outcomes.
- B. Correct.
Input data validation checks are essential to ensure the quality and accuracy of the data being processed. Without validation, corrupt or incomplete data can lead to discrepancies in the output.
- C. Incorrect.
Using AWS Glue job parameters is helpful for flexibility in ETL processing but does not directly address the data consistency or troubleshooting issues in this scenario.
- D. Correct.
Enabling CloudWatch Logs for the AWS Glue job provides visibility into errors or anomalies during processing, making it easier to troubleshoot issues and ensure repeatable outcomes.
- E. Incorrect.
Switching to AWS Lambda for ETL processing is not a direct solution to the problem of discrepancies in the current pipeline and may introduce additional complexity.