Databricks Data Engineer Professional Question 79
Single answerYou are working on a streaming pipeline in Databricks that processes real-time user activity data, joining it with a static Delta table containing user profile information. You observe that your stream-static join is causing high latency due to excessive state management. Which approach should you take to optimize the state information while ensuring accurate join results?
- A
Use a watermark on the streaming data to limit the amount of state being stored.
- B
Enable Delta Lake's ZORDER on the static table to optimize file access during the join.
- C
Reduce the checkpoint interval to decrease the size of the state stored on disk.
- D
Filter the static Delta table based on streaming data keys to reduce the size of the join.
Show answer and explanation
Correct answer: D
Explanation
In stream-static joins, the state size and performance can be impacted by the volume of data involved in the join. By filtering the static Delta table to only include relevant keys based on the streaming data, you can significantly reduce the amount of data in the join operation, thereby optimizing state management and improving the latency of the streaming pipeline. Other options like watermarking or ZORDER do not directly address the challenge in this scenario.
- A. Incorrect.
Using a watermark is useful for stream-stream joins to limit late data, but it does not optimize state management for a stream-static join.
- B. Incorrect.
ZORDER improves query performance on Delta tables by optimizing file access, but it does not directly impact the state size in stream-static joins.
- C. Incorrect.
Reducing the checkpoint interval controls how often progress is saved but does not optimize state size or join performance.
- D. Correct.
Filtering the static Delta table based on streaming data keys reduces the size of the static data involved in the join, which directly minimizes the state size and improves performance.