Google Professional Data Engineer Question 24
Select 3Google Cloud PlatformYou are working as a Data Engineer for a company that processes large volumes of customer interaction data. The raw data contains inconsistent date formats, missing values, and redundant columns that need to be cleaned before running analytics. You want to build a scalable and repeatable data preparation pipeline. Which combination of tools should you use to prepare and clean the data efficiently?
- A
Use Cloud Dataprep to visually identify and clean data inconsistencies, and then export the cleaned data to Cloud Storage.
- B
Write a custom Python script to clean and transform the data locally before uploading it to BigQuery.
- C
Use Cloud Dataflow to create a scalable pipeline to handle large-scale transformations, such as standardizing date formats and removing redundant columns.
- D
Use Cloud Data Fusion to design and orchestrate a data transformation pipeline using pre-built connectors and transformations.
- E
Manually clean the data using a spreadsheet application and upload the cleaned file to BigQuery.
Show answer and explanation
Correct answers: A, C, D
Explanation
To build a scalable and repeatable data preparation pipeline, using Google Cloud-native tools like Cloud Dataprep, Cloud Dataflow, and Cloud Data Fusion is the best approach. These tools are designed for handling large-scale data processing, transformations, and integrations efficiently. While manual or custom coding techniques can work for small datasets or specific use cases, they lack scalability, repeatability, and integration with other Google Cloud services.
- A. Correct.
Cloud Dataprep is a great choice for visually identifying and cleaning data inconsistencies. It allows you to interactively clean data and export it to Cloud Storage or BigQuery, making the process repeatable and efficient.
- B. Incorrect.
While a custom Python script could work, it is not an efficient or scalable solution for large datasets. It also lacks the repeatability and integration that Google Cloud-native tools provide.
- C. Correct.
Cloud Dataflow is an ideal choice for creating scalable pipelines for complex transformations like standardizing date formats and handling large-scale data processing.
- D. Correct.
Cloud Data Fusion provides a no-code/low-code interface with pre-built connectors and transformations, allowing you to design and orchestrate data pipelines without heavy coding.
- E. Incorrect.
Manually cleaning data in a spreadsheet is not scalable or repeatable, especially for large datasets. This approach is error-prone and does not align with best practices for data engineering.