Google Professional Data Engineer Question 275
Select 4Google Cloud PlatformAs a data engineer, you are designing a data pipeline on Google Cloud to ingest and process data from multiple sources. One of the sources occasionally sends corrupted or incomplete data. To ensure data quality and prevent pipeline failures, what steps should you include in your design?
- A
Implement data validation rules in Cloud Dataflow to filter out corrupted or incomplete records during processing.
- B
Use Cloud Storage Object Versioning to keep track of previous versions of ingested data in case of corruption.
- C
Set up BigQuery table partitioning to automatically discard corrupted or incomplete data during query execution.
- D
Create automated alerts in Cloud Monitoring to notify the team about anomalies in data ingestion patterns.
- E
Enable schema enforcement in BigQuery to reject records that do not conform to the expected schema.
Show answer and explanation
Correct answers: A, B, D, E
Explanation
To prepare for data corruption and handle missing data, it's crucial to include multiple strategies in your data pipeline design. Data validation in Cloud Dataflow ensures that only clean data proceeds through the pipeline. Cloud Storage Object Versioning allows recovery in case of corrupted data. Cloud Monitoring alerts provide visibility into potential issues, while schema enforcement in BigQuery ensures data integrity during ingestion. BigQuery table partitioning, while useful for performance, is not relevant to managing corrupted or incomplete data.
- A. Correct.
Implementing data validation rules in Cloud Dataflow is a best practice to filter out corrupted or incomplete records before they are processed further in the pipeline.
- B. Correct.
Using Cloud Storage Object Versioning helps you retain older, uncorrupted versions of data, which can be useful for recovery or analysis during corruption incidents.
- C. Incorrect.
BigQuery table partitioning is not used to discard corrupted or incomplete data. Its primary purpose is to optimize query performance and data organization.
- D. Correct.
Automated alerts in Cloud Monitoring can provide real-time notifications about unusual patterns in data ingestion, allowing the team to investigate and address issues proactively.
- E. Correct.
Schema enforcement in BigQuery ensures that only records conforming to the expected schema are ingested, reducing the risk of corrupted or unexpected data in your tables.