Databricks Data Engineer Professional Question 146
Select 3You are designing a multiplex bronze table to support multiple streaming sources in a Delta Lake architecture. Which of the following design considerations will help you avoid common pitfalls when productionalizing these streaming workloads?
- A
Use a single schema for all incoming streams to simplify processing downstream.
- B
Add a source-specific column to identify the origin of each record.
- C
Partition the table by a high-cardinality column to improve query performance.
- D
Implement schema evolution to handle changing data structures in the streams.
- E
Avoid overwriting the bronze table to ensure data lineage is preserved.
Show answer and explanation
Correct answers: B, D, E
Explanation
Designing a multiplex bronze table involves ensuring the system can handle multiple sources, evolving schemas, and preserving data lineage. Adding a source-specific column (e.g., 'source_name') allows tracking the origin of data, while implementing schema evolution ensures the system can adapt to changing data structures. Additionally, avoiding overwrites in the bronze table preserves raw data for lineage and troubleshooting, which is critical for reliable productionalized streaming workloads.
- A. Incorrect.
Using a single schema for all incoming streams may simplify downstream processing but can lead to schema conflicts or loss of source-specific information when dealing with diverse data sources.
- B. Correct.
Adding a source-specific column helps track the origin of records, which is crucial for debugging and downstream transformations in a multiplex bronze table.
- C. Incorrect.
Partitioning by a high-cardinality column can lead to small files and poor query performance. Instead, consider partitioning by a low-cardinality column, such as ingestion date.
- D. Correct.
Schema evolution allows handling changes in data structure without breaking the pipeline, which is essential for production-grade streaming workloads.
- E. Correct.
Avoiding overwriting ensures that the original raw data is preserved for audit, reprocessing, or debugging purposes, which aligns with best practices for maintaining data lineage in a bronze table.