Databricks Data Engineer Professional Question 128
Single answerYou are working on a data engineering pipeline in Databricks, where raw IoT sensor data is ingested into a Bronze table. Your team plans to promote this data to a Silver table. What is the primary objective of performing data transformations during this promotion step?
- A
To filter and clean the data by removing duplicates or corrupted records.
- B
To apply advanced machine learning models to predict future sensor values.
- C
To enrich the data with additional business logic or external datasets.
- D
To ensure the data is stored in a format optimized for low-latency queries.
Show answer and explanation
Correct answer: A
Explanation
The main objective of the Bronze-to-Silver transformation is to clean and filter raw data to ensure its quality and reliability for downstream processing. This step focuses on removing duplicates, handling corrupted records, and standardizing the data structure, preparing it for further enrichment or analysis in subsequent stages.
- A. Correct.
This is correct. The primary goal of the Bronze-to-Silver transformation is to filter, clean, and prepare the data for more meaningful analysis by removing duplicates or corrupted records.
- B. Incorrect.
This is incorrect. While machine learning may be applied later in the pipeline, it is not the primary focus of the Bronze-to-Silver transformation stage.
- C. Incorrect.
This is incorrect. Enriching data with business logic or external datasets is typically done in the Silver-to-Gold transformation stage, not during the Bronze-to-Silver promotion.
- D. Incorrect.
This is incorrect. Optimizing data for low-latency queries is more relevant to the Gold layer, where data is prepared for reporting and analytics.