Databricks Data Engineer Professional Question 148
Select 3You are tasked with designing a bronze table in Databricks to handle streaming data from multiple sources efficiently. To avoid common pitfalls such as schema evolution issues and performance degradation, which of the following approaches should you implement?
- A
Use metadata columns such as 'source' and 'ingestion_time' to distinguish data from different streams.
- B
Ingest data from all sources into a single table without any metadata columns to simplify the pipeline.
- C
Partition the bronze table based on a high-cardinality column, such as 'user_id', to optimize query performance.
- D
Use a write-ahead log (WAL) for streaming ingestion to ensure data consistency.
- E
Validate and enforce schema consistency at the source level to avoid downstream schema drift.
Show answer and explanation
Correct answers: A, D, E
Explanation
When designing a multiplex bronze table for streaming workloads, it is essential to address common challenges such as data origin tracking, schema evolution, and data consistency. Using metadata columns helps in tracking the source and ingestion time of data. Employing a write-ahead log ensures consistency and recovery in case of failures. Finally, validating schema consistency at the source level prevents downstream issues, making the pipeline more robust and maintainable.
- A. Correct.
Using metadata columns like 'source' and 'ingestion_time' helps identify the origin of data and its freshness, making debugging and data organization easier.
- B. Incorrect.
Ingesting data into a single table without metadata columns leads to challenges in tracking data origin and managing schema evolution, which is not recommended.
- C. Incorrect.
Partitioning on a high-cardinality column like 'user_id' is not advised, as it can lead to performance degradation due to excessive small files and inefficient queries.
- D. Correct.
Using a write-ahead log (WAL) ensures data consistency and helps handle failures during streaming ingestion, which is critical for production-grade pipelines.
- E. Correct.
Validating and enforcing schema consistency at the source level prevents schema drift from propagating downstream, ensuring stable pipeline operations.