Databricks Data Engineer Professional Question 84
Select 3You are designing a streaming pipeline in Databricks that performs a stream-static join between a streaming dataset of user activity logs and a static Delta table of user profiles. The streaming data has a high event volume, and the join key is user_id. To optimize the state management of this join and prevent excessive memory usage, which of the following actions should you take?
- A
Use a watermarked event time column in the streaming dataset to define state retention.
- B
Optimize the Delta table containing user profiles by running a ZORDER operation on the user_id column.
- C
Disable state checkpointing to avoid overhead during the streaming query execution.
- D
Enable trigger-based processing with a fixed micro-batch interval to control memory consumption.
- E
Ensure the static Delta table is read with caching enabled to improve join performance.
Show answer and explanation
Correct answers: A, B, E
Explanation
Stream-static joins in Databricks require careful tuning of state management to handle high event volumes efficiently. Using watermarks on the streaming data ensures that state is retained only for relevant records, which prevents excessive memory usage. Optimizing the Delta table with ZORDER helps in efficient lookups during the join, and caching the static dataset further improves performance by reducing I/O overhead. These techniques together ensure a scalable and performant stream-static join.
- A. Correct.
Using a watermarked event time column ensures that state is retained only for relevant keys, reducing memory usage in stream-static joins. This is a best practice for managing state efficiently.
- B. Correct.
Optimizing the Delta table with ZORDER on the join key (user_id) improves lookup efficiency during the join process, reducing the overhead of scanning the static data.
- C. Incorrect.
Disabling state checkpointing is not recommended as it compromises fault tolerance, and it does not help with optimizing state management.
- D. Incorrect.
While trigger-based processing can help control batch intervals, it does not directly optimize state retention or stream-static join performance.
- E. Correct.
Enabling caching for the static Delta table ensures faster lookups during the join operation, which reduces the overall query execution time and resource usage.