Databricks Data Engineer Professional Question 82
Select 3You are working with a streaming dataset of user transactions and a static dataset of user profiles stored in Delta Lake. You need to perform a stream-static join to enrich the transactions with user profile information and ensure optimal performance. Which of the following steps should you take to efficiently manage the state information during the join?
- A
Configure a watermark on the streaming query to remove old state information.
- B
Partition the static Delta Lake table by the join key to optimize shuffle operations.
- C
Use a broadcast join to reduce the amount of state information stored on the stream.
- D
Ensure the static Delta Lake table is compacted using 'OPTIMIZE' before performing the join.
- E
Set the state store timeout to a high value to retain all state information for troubleshooting purposes.
Show answer and explanation
Correct answers: A, B, D
Explanation
To efficiently manage the state information when performing stream-static joins, you should follow best practices such as using watermarks to control state retention, partitioning the static Delta Lake table by the join key to minimize shuffle, and compacting the static Delta Lake table to optimize query performance. Avoid retaining excessive state information or using unsuitable join strategies like broadcast joins, as these can negatively affect performance.
- A. Correct.
Configuring a watermark allows older data to be removed from the state store, which helps reduce memory usage during stream-static joins. This is a best practice for managing state efficiently.
- B. Correct.
Partitioning the static Delta Lake table by the join key reduces shuffle operations during the join, improving performance and scalability.
- C. Incorrect.
Broadcast joins are not suitable for stream-static joins because the static table is usually large and cannot be efficiently broadcasted.
- D. Correct.
Compacting the static Delta Lake table with 'OPTIMIZE' ensures fewer small files, reducing query latency during the stream-static join.
- E. Incorrect.
Setting the state store timeout to a high value retains unnecessary state information, leading to increased memory usage and degraded performance.