Databricks Data Engineer Professional Question 152
Select 3You are designing a real-time data pipeline in Databricks to process streaming data from multiple bronze tables into a single silver table. The bronze tables are highly multiplexed, with varied schemas and high ingestion rates. Which of the following best practices should you implement to ensure efficient and reliable processing of the streaming data?
- A
Use schema evolution to dynamically handle changes in the schemas of bronze tables.
- B
Use a watermark to drop late data and prevent state build-up in the stream.
- C
Combine all bronze table streams into a single streaming query for simplicity.
- D
Enable checkpointing to maintain state and recover from failures.
- E
Apply transformations and filtering logic in bronze tables to reduce data size before reaching the silver table.
Show answer and explanation
Correct answers: B, D, E
Explanation
When streaming data from multiplex bronze tables, best practices include using watermarks to manage late-arriving data, enabling checkpointing to ensure fault tolerance, and applying transformations or filtering logic early to reduce data size and complexity. These practices optimize performance, improve reliability, and simplify downstream processing. However, combining streams into a single query or over-relying on schema evolution in multiplexed scenarios can lead to performance bottlenecks and operational challenges.
- A. Incorrect.
While schema evolution can be helpful, relying solely on it across highly multiplexed tables can introduce complexity and errors. It's better to normalize or pre-validate schemas before streaming to silver.
- B. Correct.
Using a watermark is a best practice to handle late-arriving data and prevent the accumulation of stale state in the streaming engine.
- C. Incorrect.
Combining all bronze table streams into a single query is not a best practice as it can lead to performance bottlenecks and increase operational complexity. Separate queries for different streams provide better manageability.
- D. Correct.
Checkpointing is critical in streaming systems to maintain state and ensure fault tolerance by allowing the system to resume processing from the last successful state.
- E. Correct.
Applying transformations and filtering logic directly on bronze tables reduces the data size and complexity before the data reaches the silver table, improving overall efficiency.