Google Professional Data Engineer Question 85
Select 3Google Cloud PlatformYour organization is processing large volumes of customer data from various sources, including CSV files, streaming data, and APIs. You notice inconsistencies in the data, such as missing values, duplicate entries, and invalid formats. You want to use Google Cloud tools to implement a data cleansing process that ensures the data is accurate and reliable before loading it into BigQuery for analytics. Which of the following steps would you include in your data cleansing pipeline?
- A
Use Cloud Dataflow to deduplicate records and fill missing values with appropriate defaults.
- B
Use Cloud Dataproc to train a machine learning model to predict missing values.
- C
Use Cloud Data Fusion to apply transformations to standardize data formats.
- D
Directly load the raw data into BigQuery and use SQL queries to fix inconsistencies during analysis.
- E
Use BigQuery's data quality functions to validate data after cleansing in Cloud Dataflow.
Show answer and explanation
Correct answers: A, C, E
Explanation
Data cleansing is a critical step in preparing data for analytics and ensures data accuracy and reliability. Cloud Dataflow is ideal for deduplication and handling missing values, while Cloud Data Fusion simplifies the process of applying transformations. BigQuery's validation functions provide an additional layer of quality assurance after the cleansing process. Training machine learning models or relying solely on SQL queries for raw data is not a recommended approach for data cleansing in this scenario.
- A. Correct.
Correct: Cloud Dataflow is a managed service for stream and batch processing, and it can be used to deduplicate records and handle missing values through transformations and aggregations.
- B. Incorrect.
Incorrect: While Cloud Dataproc can be used for machine learning tasks, training a model to predict missing values is not typically part of a standard data cleansing pipeline.
- C. Correct.
Correct: Cloud Data Fusion provides a visual interface for building ETL pipelines, which can be used to standardize data formats and apply transformation rules.
- D. Incorrect.
Incorrect: Loading raw data directly into BigQuery without cleansing can lead to poor data quality during analysis. Cleansing should occur before loading the data.
- E. Correct.
Correct: BigQuery has built-in functions for data validation, such as checking for nulls or invalid formats. These can be used to verify the quality of data after cleansing in Cloud Dataflow.