Google Professional Data Engineer Question 36
Single answerGoogle Cloud PlatformYou are designing a data pipeline in Google Cloud to process daily transactional data from multiple retail stores. Before loading this data into BigQuery, you need to ensure the data meets specific quality standards, such as ensuring all required fields are present, dates are in the correct format, and numerical values are within acceptable ranges. Which service or feature would you use to implement these validation checks efficiently?
- A
Cloud Dataflow with Apache Beam data validation transforms
- B
BigQuery's built-in data validation rules
- C
Cloud Storage Object Metadata
- D
Cloud Pub/Sub Message Attributes
Show answer and explanation
Correct answer: A
Explanation
Data validation is a critical step to ensure data quality before loading it into a data warehouse like BigQuery. Cloud Dataflow, combined with Apache Beam, provides the flexibility to implement custom validation logic, such as checking for required fields, validating formats, and enforcing value ranges. This capability makes Cloud Dataflow the most suitable choice for the described scenario.
- A. Correct.
Cloud Dataflow with Apache Beam data validation transforms is the correct option, as it allows you to implement custom data validation logic programmatically during the data processing stage.
- B. Incorrect.
BigQuery does not have built-in data validation rules; it is primarily used for data storage and analytics. Validation needs to occur before loading data into BigQuery.
- C. Incorrect.
Cloud Storage Object Metadata is used to store metadata about objects in Cloud Storage, but it does not provide any data validation functionality.
- D. Incorrect.
Cloud Pub/Sub Message Attributes are used to attach metadata to messages in Pub/Sub, but they are not designed for data validation.