Databricks Data Engineer Professional Question 124
Select 3A data engineering team is designing a Lakehouse architecture for a retail company. They are using Databricks to ingest and store raw transactional data in a Bronze table, clean and enrich it in a Silver table, and finally aggregate it for reporting in a Gold table. Which of the following practices should the team follow when designing the Silver table to ensure optimal performance and usability?
- A
Use Delta Lake for the Silver table to enable ACID transactions and time travel.
- B
Store the Silver table in Parquet format instead of Delta Lake to improve query speed.
- C
Remove duplicates and clean the data to ensure accuracy and consistency.
- D
Include only aggregated metrics in the Silver table to reduce data size.
- E
Partition the Silver table based on frequently queried columns to optimize read performance.
Show answer and explanation
Correct answers: A, C, E
Explanation
When designing the Silver table in a Lakehouse architecture, it is important to use Delta Lake for its advanced features, clean and de-duplicate the data for accuracy, and partition the table for optimized read performance. These practices ensure that the Silver table is reliable, performant, and ready for downstream processing. Avoid practices like storing only aggregated metrics or using plain Parquet, as these would limit flexibility and functionality in the pipeline.
- A. Correct.
Delta Lake is a critical component of the Lakehouse architecture as it provides ACID compliance, versioning, and time travel, which are essential for reliable pipeline operations. Using Delta Lake for the Silver table ensures data integrity and usability.
- B. Incorrect.
While Parquet is a columnar storage format, Delta Lake extends Parquet by providing additional features like ACID transactions and indexing. Using only Parquet would not support the advanced features required for the Silver table.
- C. Correct.
The Silver table is typically meant to provide clean, de-duplicated, and enriched data for downstream processing. Removing duplicates and cleaning the data is a key step in this stage.
- D. Incorrect.
Aggregated metrics are generally stored in the Gold table, not the Silver table. The Silver table should contain enriched but detailed data to allow flexibility for various downstream use cases.
- E. Correct.
Partitioning the Silver table on frequently queried columns is a best practice to improve read performance, especially for large datasets, as it helps Databricks read only the necessary partitions.