DEA-C01 Question 352
Select 3You are managing a data pipeline in AWS Glue that processes large volumes of streaming data from an Amazon Kinesis Data Stream. Recently, the pipeline has started to experience delays, and you need to investigate and resolve the issue. Which of the following actions should you take to maintain and monitor the pipeline effectively?
- A
Enable AWS Glue job metrics in Amazon CloudWatch to monitor execution time and memory usage.
- B
Increase the number of shards in the Kinesis Data Stream to improve data throughput.
- C
Use AWS Glue's built-in retry mechanism to automatically handle transient failures in the pipeline.
- D
Inspect the AWS Glue job logs in Amazon CloudWatch for detailed error messages and debugging information.
- E
Enable auto-scaling for the AWS Glue job to dynamically adjust resources based on the workload.
Show answer and explanation
Correct answers: A, B, D
Explanation
To effectively maintain and monitor the data pipeline, you must first gather relevant metrics and logs to understand the root cause of delays. Enabling AWS Glue job metrics in CloudWatch and inspecting logs provides visibility into the performance and potential errors. If the Kinesis Data Stream is a bottleneck, increasing the number of shards can improve throughput. While the retry mechanism is useful for transient failures, and auto-scaling sounds beneficial, they do not directly address the issue in this scenario.
- A. Correct.
Enabling AWS Glue job metrics in CloudWatch is essential for monitoring execution time, memory usage, and other performance-related metrics, helping you identify bottlenecks in the pipeline.
- B. Correct.
Increasing the number of shards in the Kinesis Data Stream can improve data throughput if the stream is becoming a bottleneck due to high data ingestion rates.
- C. Incorrect.
AWS Glue's built-in retry mechanism is useful for handling transient errors, but it does not directly help in diagnosing or resolving pipeline delays.
- D. Correct.
Inspecting AWS Glue job logs in CloudWatch provides detailed error messages and debugging information, which are critical for identifying the root cause of pipeline inefficiencies.
- E. Incorrect.
AWS Glue does not support auto-scaling in the same way as other AWS services like Lambda or ECS. You would need to manually adjust job parameters like allocated DPUs.