Google Professional Data Engineer Question 35
Select 3Google Cloud PlatformYou are designing a data processing pipeline on Google Cloud, and you need to validate incoming data to ensure it conforms to specific schema requirements before it is processed further. Which of the following approaches would help you implement data validation effectively within your pipeline?
- A
Use Dataflow with a custom validation function to enforce schema rules during data transformation.
- B
Store incoming data in BigQuery and use SQL queries to validate the schema post-ingestion.
- C
Leverage Cloud Data Fusion's Wrangler to apply data quality rules and schema validation before processing.
- D
Use a Pub/Sub topic to directly enforce schema validation during message publication.
- E
Implement schema validation in Cloud Functions triggered by Cloud Storage events.
Show answer and explanation
Correct answers: A, C, E
Explanation
Data validation is a critical step in ensuring that the data conforms to expected schema rules before it is processed further. Approaches like using Dataflow with custom validation functions, leveraging Cloud Data Fusion's Wrangler, or implementing schema validation in Cloud Functions triggered by Cloud Storage events help ensure schema compliance at different stages of the pipeline. However, relying on post-ingestion validation in BigQuery or using Pub/Sub for schema validation is either inefficient or unsupported.
- A. Correct.
This is correct because Dataflow allows you to define custom validation logic as part of your data transformation pipeline, ensuring data conforms to schema rules early in the process.
- B. Incorrect.
This is not ideal because validating data post-ingestion into BigQuery may result in processing invalid data, leading to inefficiencies and potential downstream issues.
- C. Correct.
This is correct because Cloud Data Fusion's Wrangler provides tools for applying data quality and schema validation rules interactively before processing.
- D. Incorrect.
This is incorrect because Pub/Sub does not have built-in schema validation features during message publication. Alternatively, schema validation would need to be implemented in downstream systems.
- E. Correct.
This is correct because Cloud Functions can be configured to trigger on Cloud Storage events (e.g., file uploads) and execute custom schema validation logic before passing the data to further stages.