DEA-C01 Question 353
Select 4You are managing a data pipeline in AWS that ingests data from Amazon Kinesis Data Streams, processes it using AWS Glue, and stores the results in an S3 bucket. Recently, you noticed delays in the pipeline, and some data is not being processed as expected. Which of the following actions should you perform to maintain and monitor the pipeline effectively?
- A
Enable Amazon CloudWatch metrics and alarms for Kinesis Data Streams to monitor throughput and shard utilization.
- B
Set up AWS Glue job bookmarks to ensure that the pipeline avoids reprocessing already processed data.
- C
Increase the size of the S3 bucket to accommodate more processed data.
- D
Use AWS Glue's 'workload partitioning' feature to distribute processing for better performance.
- E
Enable AWS Glue job metrics and logs in Amazon CloudWatch to monitor job execution and troubleshoot errors.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To maintain and monitor a data pipeline effectively, you must address potential bottlenecks and enhance visibility into the pipeline's components. Monitoring Kinesis Data Streams and AWS Glue metrics via CloudWatch helps identify issues in data ingestion and processing. Utilizing features like Glue job bookmarks and workload partitioning ensures efficient and scalable data processing. Simply expanding storage, such as an S3 bucket, does not resolve pipeline performance or monitoring issues.
- A. Correct.
Correct. Enabling CloudWatch metrics and alarms for Kinesis Data Streams allows you to monitor shard utilization and throughput, identifying bottlenecks in the ingestion phase.
- B. Correct.
Correct. AWS Glue job bookmarks help prevent reprocessing of data, ensuring the pipeline operates efficiently without redundant processing.
- C. Incorrect.
Incorrect. Increasing the size of the S3 bucket will not directly resolve pipeline delays or unprocessed data issues. Storage space is unrelated to the pipeline's performance.
- D. Correct.
Correct. AWS Glue workload partitioning allows you to distribute data processing tasks across multiple workers, improving performance for large datasets.
- E. Correct.
Correct. Enabling AWS Glue job metrics and logs in CloudWatch provides visibility into job execution, helping you diagnose and resolve pipeline issues.