Databricks Data Engineer Professional Question 151
Select 3You are tasked with designing a streaming pipeline in Databricks to process data from multiple bronze tables into a consolidated silver table. These bronze tables contain high-volume, semi-structured data from different sources. To ensure the pipeline follows best practices and maintains data quality, what steps should you take?
- A
Use a watermark to handle late-arriving data and prevent duplicate processing.
- B
Write directly into the silver table without performing schema validation or deduplication.
- C
Implement a structured streaming query for each bronze table that includes schema enforcement and data cleansing.
- D
Avoid checkpointing to reduce storage overhead and improve performance.
- E
Consolidate data from multiple bronze tables into a single stream before writing to the silver table.
Show answer and explanation
Correct answers: A, C, E
Explanation
When streaming data from multiple bronze tables, it is crucial to follow best practices to maintain data quality and ensure efficient processing. This includes using watermarks to handle late-arriving data, implementing schema enforcement and data cleansing for each source, and consolidating data streams for consistent processing. Checkpointing is essential for fault tolerance and should not be skipped. Writing directly to the silver table without validation or deduplication is not recommended as it can lead to data quality issues.
- A. Correct.
Using a watermark is essential in streaming pipelines to handle late-arriving data and prevent reprocessing of older data, ensuring accurate and efficient processing.
- B. Incorrect.
Skipping schema validation or deduplication can lead to data quality issues, such as inconsistent schema or duplicate records, which should be avoided in a well-designed streaming pipeline.
- C. Correct.
Implementing structured streaming queries for each bronze table with schema enforcement and data cleansing ensures that data is transformed and validated before being written to the silver table, improving overall data quality.
- D. Incorrect.
Checkpointing is a critical component of stateful streaming operations that ensures fault tolerance and exactly-once processing. Avoiding checkpointing may lead to data loss or reprocessing in case of failures.
- E. Correct.
Consolidating data from multiple bronze tables into a single stream before writing to the silver table helps to streamline processing and ensures a unified view of the data.