Databricks Data Engineer Professional Question 87
Single answerYou are tasked with designing a Databricks Structured Streaming pipeline to process real-time transaction data from a Kafka source. The pipeline also needs to enrich the streaming data with static reference data stored in a Delta table. Which of the following approaches is correct for implementing a stream-static join in this scenario?
- A
Use a
JOINoperation between the streaming DataFrame and the static Delta table directly, ensuring that the static Delta table fits in memory. - B
Use a
JOINoperation between the streaming DataFrame and the static Delta table without any constraints on memory requirements. - C
First broadcast the static Delta table and then use a
JOINoperation with the streaming DataFrame. - D
Use
mapGroupsWithStateto join the streaming DataFrame with the static Delta table.
Show answer and explanation
Correct answer: A
Explanation
Stream-static joins in Databricks Structured Streaming allow you to enrich real-time streams with static reference data. For efficient processing, the static table must fit in memory. Databricks optimizes these operations automatically, provided the static data is appropriately sized and read as a Delta table.
- A. Correct.
This is correct because a stream-static join requires the static data to fit in memory for efficient processing. Databricks can handle this efficiently when the static table is reasonably sized.
- B. Incorrect.
This is incorrect because if the static Delta table is too large to fit in memory, the join operation may fail or perform poorly.
- C. Incorrect.
This is incorrect because broadcasting is not necessary for stream-static joins in Databricks Structured Streaming. Broadcast joins are used in other scenarios, such as performance optimization for small tables in batch queries.
- D. Incorrect.
This is incorrect because
mapGroupsWithStateis used for stateful processing and not for implementing stream-static joins.