Databricks Data Engineer Professional Question 145
Select 3You are designing a bronze table to ingest streaming data from multiple sources, such as IoT devices, web logs, and transactional systems. To avoid common pitfalls and ensure scalability and maintainability of the multiplex bronze table, which of the following best practices should you follow?
- A
Include a 'source' column to differentiate data from various sources.
- B
Use a single schema for all sources, even if their data structures differ.
- C
Partition the table by an ingestion timestamp column for efficient querying.
- D
Avoid storing raw data in the bronze table and instead pre-clean the data during ingestion.
- E
Implement schema evolution to handle changes in source data schemas.
Show answer and explanation
Correct answers: A, C, E
Explanation
When designing a multiplex bronze table, it is crucial to include metadata (like the 'source' column) to distinguish data from different sources. Partitioning by an ingestion timestamp improves query performance and scalability. Schema evolution ensures the table can handle changes in source data schemas over time. Following these best practices helps avoid common pitfalls, such as schema conflicts and inefficient querying, ensuring the streaming workload is production-ready.
- A. Correct.
Including a 'source' column helps identify the origin of the data, making it easier to process and troubleshoot downstream, especially when dealing with multiple sources.
- B. Incorrect.
Using a single schema for all sources is not a recommended best practice when their data structures differ, as it can lead to schema conflicts and data corruption during ingestion.
- C. Correct.
Partitioning the table by an ingestion timestamp ensures efficient querying and improves the performance of downstream processing.
- D. Incorrect.
Avoiding raw data storage in the bronze table goes against the principle of a bronze table, which is meant to store unaltered raw data for traceability and reproducibility.
- E. Correct.
Implementing schema evolution allows the bronze table to adapt to changes in source data schemas without disrupting the streaming ingestion process.