Databricks Data Engineer Professional Question 147
Select 2You are designing a multiplex bronze table to handle multiple streaming workloads in a Databricks Lakehouse environment. Which of the following best practices should you follow to ensure scalability, avoid schema evolution issues, and handle diverse data sources effectively?
- A
Use a unified schema across all incoming streams to ensure consistency and avoid schema evolution conflicts.
- B
Partition the table by a frequently queried column, such as event type or source system, to improve query performance and scalability.
- C
Include metadata columns, such as ingestion timestamp and source system, to track data provenance and support downstream processing.
- D
Write each stream to a separate bronze table to simplify stream management and avoid conflicts.
- E
Enable schema auto-merge to automatically handle schema evolution from all streaming sources.
Show answer and explanation
Correct answers: B, C
Explanation
When designing a multiplex bronze table, the goal is to consolidate data from multiple streams into a single table while ensuring scalability and avoiding schema evolution pitfalls. Partitioning the table by a frequently queried column improves performance, while including metadata columns provides critical context for downstream processing. Using a unified schema or enabling schema auto-merge can lead to issues with schema conflicts or unintended changes, making them less ideal approaches for production-grade systems.
- A. Incorrect.
Using a unified schema across all streams can be overly restrictive when dealing with diverse data sources, potentially leading to dropped or incompatible data if the schema doesn't match perfectly. This is not a recommended practice for handling multiplex workloads.
- B. Correct.
Partitioning the table by a frequently queried column, such as event type or source system, improves query performance and scalability by allowing efficient data retrieval and reducing the need to scan the entire dataset.
- C. Correct.
Including metadata columns like ingestion timestamp and source system ensures data provenance, supports debugging, and enables downstream processing to filter and process data based on its origin or ingestion time.
- D. Incorrect.
Writing each stream to a separate bronze table negates the purpose of a multiplex bronze table, which is designed to consolidate data from multiple streams into a single table for simplicity and scalability.
- E. Incorrect.
Enabling schema auto-merge can lead to unintended schema changes, such as data type mismatches or column order changes, which can break downstream workflows. Schema management should be handled explicitly and cautiously.