Databricks Data Engineer Professional Question 149
Select 2You are designing a multiplex bronze table to handle production-grade streaming workloads in Databricks. Which of the following approaches will help you avoid common pitfalls such as schema evolution issues, data duplication, and high storage costs?
- A
Use a single bronze table for all streaming sources regardless of their schema.
- B
Partition the bronze table by a meaningful column, such as ingestion date, to optimize query performance and data management.
- C
Use a Delta Lake table with append-only mode to prevent accidental overwrites and ensure immutability of raw data.
- D
Define a schema for each streaming source and merge the schemas into the bronze table dynamically at runtime.
- E
Store raw and processed data in the same bronze table to simplify data management.
Show answer and explanation
Correct answers: B, C
Explanation
To design a production-grade multiplex bronze table, it is essential to optimize for performance and reliability. Partitioning the table by a meaningful column like ingestion date helps with query optimization and data organization. Additionally, using a Delta Lake table in append-only mode ensures data immutability and prevents accidental overwrites, making it suitable for raw data storage. Avoiding schema evolution issues and maintaining clear separation between raw and processed data are also critical best practices.
- A. Incorrect.
Using a single bronze table for all streaming sources regardless of their schema can lead to schema evolution issues and makes it difficult to manage data from diverse sources effectively.
- B. Correct.
Partitioning the bronze table by a meaningful column like ingestion date helps optimize query performance and manage data efficiently, which is critical for production workloads.
- C. Correct.
Using a Delta Lake table with append-only mode ensures the immutability of raw data and avoids accidental overwrites, which is a best practice for bronze table design.
- D. Incorrect.
Dynamically merging schemas at runtime can introduce schema evolution issues and complicate debugging. Explicit schema definitions are preferred for better control.
- E. Incorrect.
Storing raw and processed data in the same bronze table violates the principle of separating raw data (bronze) from transformed data (silver/gold) and increases complexity.