Databricks Data Engineer Professional Question 156
Select 3You are building a streaming pipeline in Databricks to process data from multiple bronze tables into a silver table. Each bronze table contains raw data from a different source, and updates to these tables occur at varying intervals. Which of the following are best practices to implement when streaming data from these multiplex bronze tables into the silver layer?
- A
Use watermarking to handle late-arriving data from the bronze tables.
- B
Perform schema evolution on the bronze tables directly to ensure compatibility with downstream processing.
- C
Use trigger-based incremental processing to balance resource utilization and data latency.
- D
Implement deduplication logic in the streaming queries to prevent duplicate records in the silver table.
- E
Read from all bronze tables simultaneously without partitioning to maximize throughput.
Show answer and explanation
Correct answers: A, C, D
Explanation
When streaming data from multiplex bronze tables, it is essential to use watermarking to manage late-arriving data, implement deduplication logic to avoid duplicates, and leverage trigger-based incremental processing to optimize resource utilization. Avoid directly modifying the schema of bronze tables and ensure efficient reads by partitioning data appropriately.
- A. Correct.
Watermarking is a best practice for handling late-arriving data in streaming pipelines. It allows you to define a threshold for how late data can be and still be processed, ensuring the pipeline remains efficient.
- B. Incorrect.
Schema evolution should not be performed directly on the bronze tables because they contain raw data. Schema changes should be handled at the silver or gold layer to ensure source data integrity.
- C. Correct.
Trigger-based incremental processing (e.g., 'Trigger.Once' or 'Trigger.AvailableNow') helps balance resource utilization and data latency, especially when dealing with multiple data sources with varying update frequencies.
- D. Correct.
Deduplication logic is crucial in streaming pipelines to ensure that duplicate records from the bronze layer do not propagate to the silver layer.
- E. Incorrect.
Reading from all bronze tables simultaneously without partitioning is not recommended as it can lead to resource contention and inefficiencies when processing large datasets.