DEA-C01 Question 303
Select 3Your company processes sales transaction data using an AWS Glue ETL job, which runs daily to populate an Amazon Redshift data warehouse. Recently, the business has reported inconsistent data in the dashboards due to missing or incomplete records. How can you maintain consistent and repeatable data processing outcomes while troubleshooting the issue?
- A
Use AWS Glue job bookmarks to ensure that the ETL job processes only new or updated data.
- B
Enable CloudWatch Logs for the AWS Glue job to monitor for errors or failed tasks during ETL execution.
- C
Switch to an on-demand AWS Glue job trigger to manually review data before every job execution.
- D
Implement Amazon Redshift's COPY command with the
COMPUPDATEoption to automatically handle schema changes. - E
Set up data quality checks using AWS Deequ or custom validation scripts to verify data completeness.
Show answer and explanation
Correct answers: A, B, E
Explanation
To maintain consistent and repeatable data processing outcomes, you should leverage AWS Glue job bookmarks to process data incrementally, enable logging to troubleshoot errors, and implement data quality checks to validate the integrity of the processed data. These practices collectively ensure that your ETL pipeline is reliable and produces accurate results for downstream analytics.
- A. Correct.
Using AWS Glue job bookmarks ensures that your ETL job processes only new or updated data, preventing duplication or missing records across job runs. This is crucial for repeatable outcomes.
- B. Correct.
Enabling CloudWatch Logs provides visibility into ETL job execution, helping troubleshoot errors or identify issues during data processing.
- C. Incorrect.
Switching to an on-demand trigger introduces manual intervention, which can lead to inconsistent execution schedules and increased operational overhead. This is not a recommended practice for maintaining repeatable outcomes.
- D. Incorrect.
While the
COMPUPDATEoption in Amazon Redshift's COPY command is useful for handling schema changes, it does not address the issue of missing or incomplete records in the ETL pipeline. - E. Correct.
Data quality checks using AWS Deequ or custom validation scripts ensure data completeness and consistency, which is a critical step to maintain repeatable and reliable outcomes.