Databricks Data Engineer Professional Question 150
Select 3You are tasked with designing a multiplex bronze table in Databricks to process streaming data from multiple sources. To ensure a production-ready design and avoid common pitfalls, which of the following strategies should you implement?
- A
Include a 'source' column to track the origin of each data record.
- B
Use a single schema for all incoming streaming sources to simplify processing.
- C
Partition the bronze table based on event time or ingestion date to optimize reads and writes.
- D
Overwrite the bronze table with new data whenever a source stream is restarted to maintain consistency.
- E
Implement schema evolution to handle new columns or data types arriving from any source.
Show answer and explanation
Correct answers: A, C, E
Explanation
When designing a multiplex bronze table, it is important to account for the diverse nature of streaming data sources and the need for scalability. Including a 'source' column ensures traceability, partitioning improves performance, and schema evolution allows for adaptability to changes in the data. Avoiding practices like overwriting data or enforcing a single schema ensures consistency and reliability in a production environment.
- A. Correct.
Including a 'source' column is essential when designing a multiplex bronze table as it allows you to track the origin of each data record, which is critical for debugging and downstream processing.
- B. Incorrect.
Using a single schema for all incoming streaming sources is not recommended because different sources may have distinct schemas, and enforcing a single schema could lead to data loss or processing errors.
- C. Correct.
Partitioning the bronze table based on event time or ingestion date helps optimize both reads and writes, especially for large-scale streaming data workloads.
- D. Incorrect.
Overwriting the bronze table with new data is NOT a best practice. This could lead to data loss and inconsistencies. Instead, append-only or merge-based patterns should be used for streaming workloads.
- E. Correct.
Implementing schema evolution is crucial for handling changes in the data structure, such as new columns or data types, without interrupting the streaming workload.