Databricks Data Engineer Professional Question 153
Select 2You are tasked with designing a streaming pipeline that reads data from multiple bronze tables in a Databricks Lakehouse environment. Each bronze table represents raw data from a specific source system, and the data needs to be processed into a unified silver table. To ensure best practices for streaming from these multiplex bronze tables, which of the following approaches should you implement?
- A
Use a single structured streaming query to read from all bronze tables simultaneously using a union operation.
- B
Process each bronze table in a dedicated structured streaming query and write to separate intermediate silver tables.
- C
Employ schema evolution to handle differing schemas across the bronze tables during the read operation.
- D
Use watermarking and appropriate checkpointing to manage late-arriving data from each bronze table.
- E
Optimize the join operations between bronze and silver tables by explicitly caching the bronze table streams.
Show answer and explanation
Correct answers: B, D
Explanation
When streaming from multiplex bronze tables, best practices include processing each table in a modular and scalable manner with dedicated streaming queries. This allows for independent schema handling and simplifies debugging. Additionally, watermarking and checkpointing are critical for maintaining data integrity and handling late-arriving data effectively. Union operations and caching are generally discouraged due to scalability and memory management concerns.
- A. Incorrect.
Using a single structured streaming query to read from all bronze tables simultaneously via a union can lead to scalability issues and operational complexity, especially when each bronze table has different schemas or data arrival patterns.
- B. Correct.
Processing each bronze table in a dedicated streaming query ensures modularity and allows for schema-specific transformations, improving scalability and maintainability.
- C. Incorrect.
While schema evolution is useful in some scenarios, it is not recommended to rely on it during the read operation for streaming from multiplex bronze tables because the schema differences can cause runtime errors or complicate processing.
- D. Correct.
Using watermarking and checkpointing is a best practice for streaming pipelines to handle late-arriving data and ensure fault tolerance in case of failures.
- E. Incorrect.
Caching streaming data is not recommended because it can lead to memory management issues and is not aligned with the principles of structured streaming, which processes data incrementally.