DEA-C01 Question 341
Select 3You are a data engineer tasked with building a data pipeline on AWS for processing customer transaction data. The dataset contains missing values, duplicate records, and inconsistent date formats. Which of the following cleansing techniques should you apply to ensure the dataset is ready for analysis?
- A
Remove duplicate records from the dataset to ensure unique data entries.
- B
Replace missing values with default or statistically derived values based on the data context.
- C
Ignore missing values and proceed with the pipeline to avoid delays in processing time.
- D
Normalize inconsistent date formats into a standard format across all records.
- E
Filter out rows containing any missing or incomplete data to ensure data quality.
Show answer and explanation
Correct answers: A, B, D
Explanation
Effective data cleansing is essential for ensuring the reliability and usability of data for downstream analysis. Removing duplicates, handling missing values appropriately, and standardizing formats are best practices that improve data quality without losing critical information or introducing bias.
- A. Correct.
Removing duplicate records is a fundamental data cleansing step to avoid inflated results or erroneous insights during data analysis.
- B. Correct.
Replacing missing values with default or statistically derived values ensures the dataset remains usable without creating gaps in the analysis.
- C. Incorrect.
Ignoring missing values can lead to inaccurate or incomplete analysis, making this an unsuitable cleansing technique in most scenarios.
- D. Correct.
Standardizing date formats is critical for maintaining consistency and ensuring the data can be accurately processed and analyzed.
- E. Incorrect.
Filtering out rows with missing data can lead to significant data loss, reducing the dataset's representativeness and potentially introducing bias.