Databricks Data Engineer Professional Question 89
Single answerYou are developing a streaming pipeline in Databricks where you need to join a stream of real-time transaction data with a static dataset containing customer information. The transaction stream is a high-volume Structured Streaming DataFrame, and the customer information is a small, static DataFrame. Which of the following approaches ensures an efficient stream-static join in this scenario?
- A
Broadcast the static customer DataFrame and perform a join with the streaming DataFrame.
- B
Cache the streaming DataFrame and perform a join with the static customer DataFrame.
- C
Persist both the streaming DataFrame and the static customer DataFrame to disk before performing the join.
- D
Join the streaming DataFrame and the static customer DataFrame directly without any optimizations.
Show answer and explanation
Correct answer: A
Explanation
In a stream-static join, the static dataset should be small enough to fit in memory and can be broadcasted to all executors. Broadcasting the static DataFrame optimizes the join by avoiding shuffling and ensuring efficient distribution. This is particularly important when dealing with high-volume, real-time streaming data.
- A. Correct.
Broadcasting the static DataFrame ensures that it is efficiently distributed across all executors in memory, enabling efficient stream-static joins. This is the recommended approach for joining a high-volume stream with a small, static dataset.
- B. Incorrect.
Caching the streaming DataFrame is not appropriate because streaming data is unbounded and continuously changing, making caching ineffective for this use case.
- C. Incorrect.
Persisting both DataFrames to disk is unnecessary and introduces extra overhead, especially for streaming workloads. It is not suitable for this scenario.
- D. Incorrect.
Joining the streaming DataFrame and the static DataFrame directly without optimizations can lead to poor performance because the static DataFrame would not be efficiently distributed across executors.