Databricks Data Engineer Professional Question 90
Select 3You are tasked with implementing a stream-static join in Databricks to merge a streaming DataFrame of retail transactions with a static lookup table containing product details. The goal is to enrich each transaction with product information. Which of the following steps are necessary to perform this join correctly?
- A
Ensure the static DataFrame is broadcasted to optimize the join.
- B
Define appropriate watermarking on the streaming DataFrame to handle late data efficiently.
- C
Use the 'join' operation with a condition that matches the keys between the streaming and static DataFrames.
- D
Persist the static DataFrame in memory to ensure performance during the join.
- E
Configure the streaming query to use 'update' output mode to avoid redundant outputs.
Show answer and explanation
Correct answers: B, C, E
Explanation
To implement a stream-static join in Databricks, you need to ensure the streaming DataFrame is properly configured with watermarking to handle late-arriving data and reduce state size. A join condition matching the keys between the streaming and static DataFrames is required to perform the enrichment. Additionally, using the 'update' output mode ensures efficiency by emitting only the latest updates, which is particularly important in streaming workloads.
- A. Incorrect.
Broadcasting is not required when performing a stream-static join in Databricks as the static DataFrame is already optimized for lookups.
- B. Correct.
Watermarking is essential for managing late-arriving data in streaming workloads and avoiding excessive state buildup.
- C. Correct.
The join operation with a proper condition is mandatory to merge the streaming DataFrame with the static DataFrame based on the keys.
- D. Incorrect.
Persisting the static DataFrame is unnecessary because static DataFrames are inherently optimized for repeated use in stream-static joins.
- E. Correct.
Using the 'update' output mode ensures that only the latest results are emitted, reducing redundant outputs in the streaming query.