Google Professional Data Engineer Question 162
Select 3Google Cloud PlatformYour company has implemented a data lake on Google Cloud Storage to store large volumes of raw data. You are tasked with setting up monitoring to ensure data quality and operational reliability. Which of the following monitoring approaches should you implement?
- A
Set up Cloud Storage bucket-level logging to track access and changes to the data.
- B
Use Cloud Monitoring to create custom metrics for data ingestion rates and alert on unexpected drops.
- C
Implement a BigQuery scheduled query to validate data schema compliance and store the results for auditing.
- D
Enable VPC Flow Logs to monitor network traffic between your on-premises systems and the data lake.
- E
Configure Pub/Sub notifications for Cloud Storage to trigger workflows for data validation upon file uploads.
Show answer and explanation
Correct answers: A, B, E
Explanation
Monitoring a data lake requires a combination of access monitoring, operational metrics, and real-time notifications. Cloud Storage bucket-level logging, custom Cloud Monitoring metrics, and Pub/Sub notifications address these needs effectively by offering visibility into access, operational reliability, and real-time validation workflows. Other options, such as schema validation in BigQuery and VPC Flow Logs, are less relevant to direct monitoring of the data lake.
- A. Correct.
Cloud Storage bucket-level logging is essential for monitoring access and changes to the data lake, which helps ensure data security and traceability.
- B. Correct.
Using Cloud Monitoring to create custom metrics for ingestion rates enables you to detect operational issues, such as unexpected drops or spikes in data ingestion.
- C. Incorrect.
While validating data schema compliance is useful for ensuring data quality, this task is better suited for downstream processing or ETL pipelines rather than as part of monitoring the data lake.
- D. Incorrect.
VPC Flow Logs monitor network traffic but are not directly related to monitoring the state, quality, or ingestion performance of the data lake.
- E. Correct.
Configuring Pub/Sub notifications for Cloud Storage allows you to trigger workflows, such as data validation processes, upon file uploads, ensuring real-time monitoring and quality checks.