Databricks Data Engineer Professional Question 184
Single answerYou are designing a data pipeline in Databricks to process customer order data. The pipeline must join the order records with a product lookup table to enrich the dataset with product details such as name, category, and price. The pipeline processes billions of records daily, and the product lookup table rarely changes (updated once a month). Which approach is most suitable to optimize performance and resource usage in this scenario?
- A
Broadcast the product lookup table to all worker nodes and perform a broadcast join.
- B
Store the product details as a Delta table and perform a shuffle join.
- C
Denormalize the data by embedding the product details directly into the order records.
- D
Cache the product lookup table in memory and perform a lookup operation for each order record.
Show answer and explanation
Correct answer: A
Explanation
Broadcast joins are ideal when one of the datasets is small and static, as in the case of the product lookup table. By broadcasting this table to all worker nodes, Databricks avoids the need for costly shuffle operations, leading to faster join performance and reduced resource consumption. Other approaches either introduce unnecessary overhead (shuffle join), increase storage costs and complexity (denormalization), or are less efficient (in-memory lookups) for this specific scenario.
- A. Correct.
Broadcasting the product lookup table is highly efficient for small, static datasets like this one. It avoids expensive shuffles by sending the small lookup table to all worker nodes, enabling faster joins.
- B. Incorrect.
Using a shuffle join with the Delta table is less efficient for this use case because it involves a costly shuffle operation, which is unnecessary for a small, rarely updated lookup table.
- C. Incorrect.
Denormalizing the data would lead to significant storage overhead and complicate updates to the product details. This approach sacrifices storage efficiency and maintainability for performance.
- D. Incorrect.
Caching the product lookup table in memory would require frequent lookups for each order record, which is less efficient compared to a broadcast join for large-scale data processing.