Databricks Data Engineer Professional Question 86
Select 2You are building a real-time data pipeline in Databricks to process clickstream data. The clickstream data is ingested into a streaming DataFrame, while user profile data is stored in a static Delta table. You need to join the streaming clickstream data with the static user profile data to enrich the incoming data with user attributes. Which of the following conditions must be met to implement this stream-static join correctly?
- A
The static table must be read using the Delta table format.
- B
The join condition must involve at least one equality condition between the streaming and static datasets.
- C
The streaming DataFrame must have a watermark defined to handle late-arriving data.
- D
The static dataset must fit into the memory of the cluster for the join to work.
- E
The streaming DataFrame must be written to a Delta table after the join is performed.
Show answer and explanation
Correct answers: B, D
Explanation
To perform a stream-static join in Databricks, the join must include at least one equality condition to allow efficient execution. Additionally, the static dataset is broadcasted during the join process, so it must fit into the memory of the cluster. While other options may be relevant for specific scenarios, they are not strict requirements for implementing a stream-static join.
- A. Incorrect.
The static table does not necessarily need to be in Delta format; it can be in other formats such as Parquet or a JDBC source.
- B. Correct.
For a stream-static join to work, there must be at least one equality condition in the join clause, as this is required for efficient execution.
- C. Incorrect.
A watermark is not mandatory for stream-static joins since the static dataset is not time-sensitive. Watermarks are generally used for stream-stream joins or aggregations to handle late-arriving data.
- D. Correct.
The static dataset must fit into the memory of the cluster because the static data is typically broadcasted to all nodes to perform the join efficiently.
- E. Incorrect.
It is not mandatory to write the result to a Delta table after the join; the output can be written in other formats or sent to other destinations.