Google Professional Data Engineer Question 252
Select 3Google Cloud PlatformYou are responsible for designing a real-time data processing pipeline on Google Cloud. The pipeline ingests data into Pub/Sub, processes it using Dataflow, and stores the results in BigQuery. Recently, you noticed an increase in pipeline latency and some data processing errors. Which actions should you take to monitor and troubleshoot the issue effectively?
- A
Enable Dataflow job logs and use Cloud Logging to analyze streaming pipeline errors.
- B
Set up an alerting policy in Cloud Monitoring to notify when Pub/Sub message backlog grows beyond a threshold.
- C
Use the BigQuery Query Execution graph to track query costs and optimize pipeline performance.
- D
Review Dataflow worker logs and CPU/memory utilization metrics using Cloud Monitoring.
- E
Disable Pub/Sub message retention to prevent outdated messages from being processed.
Show answer and explanation
Correct answers: A, B, D
Explanation
Monitoring and troubleshooting processes in a real-time data pipeline involve analyzing component-specific logs (Dataflow, Pub/Sub), setting up alerts for key metrics like message backlog, and reviewing resource utilization. These actions help identify and resolve latency and error issues within the pipeline efficiently. However, disabling message retention or focusing solely on downstream components like BigQuery without addressing upstream issues may not effectively resolve pipeline problems.
- A. Correct.
Correct: Enabling Dataflow job logs and analyzing them with Cloud Logging helps identify errors and bottlenecks in the streaming pipeline.
- B. Correct.
Correct: Monitoring the Pub/Sub message backlog is crucial for identifying delays in message processing, which could contribute to increased latency.
- C. Incorrect.
Incorrect: While the BigQuery Query Execution graph is useful for optimizing query performance, it does not directly address monitoring or troubleshooting pipeline latency and errors.
- D. Correct.
Correct: Reviewing worker logs and resource utilization in Cloud Monitoring helps identify resource bottlenecks and misconfigurations in your Dataflow pipeline.
- E. Incorrect.
Incorrect: Disabling message retention in Pub/Sub could result in data loss and does not help in troubleshooting or monitoring pipeline issues.