Google Professional Data Engineer Question 34
Single answerGoogle Cloud PlatformYour team is designing a data pipeline to ingest and process customer transaction data using Google Cloud. You need to ensure that the data conforms to specific quality standards (e.g., valid timestamps, non-null fields, and proper data types) before it is processed further. Which Google Cloud feature or tool can you use to implement automated data validation in the pipeline?
- A
Dataflow with Apache Beam's Schema Transform
- B
BigQuery scheduled queries
- C
Cloud Pub/Sub message attributes
- D
Cloud Storage Object Lifecycle Management
Show answer and explanation
Correct answer: A
Explanation
Data validation is a critical requirement for ensuring data quality in pipelines. Apache Beam's Schema Transform, which can be used with Dataflow, provides a structured way to enforce schema constraints and validate data during ingestion or transformation. This ensures that only valid, high-quality data is processed further, reducing downstream errors.
- A. Correct.
Dataflow with Apache Beam's Schema Transform is the correct option because it allows you to define and enforce schema validation rules in your data pipeline. You can specify constraints such as data types, required fields, and custom validation logic.
- B. Incorrect.
BigQuery scheduled queries are used for automating query execution but are not designed for data validation. They are focused on querying and transforming data, not verifying its quality during ingestion.
- C. Incorrect.
Cloud Pub/Sub message attributes can help in metadata tagging or filtering messages, but they do not provide data validation capabilities for ensuring quality standards in your pipeline.
- D. Incorrect.
Cloud Storage Object Lifecycle Management is used for managing object lifecycle policies (e.g., deleting or archiving old objects) and is unrelated to data validation.