Databricks Data Engineer Professional Question 88
Select 3You are tasked with implementing a stream-static join in a Databricks notebook. You have a streaming DataFrame that contains real-time sales transactions, and a static DataFrame that contains product details (e.g., product ID, product name, and price). The goal is to enrich the streaming sales data with product details based on the 'product_id' column. Which of the following approaches are required to correctly perform this stream-static join?
- A
Ensure the static DataFrame is broadcast to optimize join performance.
- B
Use a watermark on the streaming DataFrame to handle late-arriving data.
- C
Join the streaming DataFrame with the static DataFrame using the 'join' method and specify the join condition.
- D
Set checkpointing for the streaming query to enable fault tolerance.
- E
Use the 'foreachBatch' method to manually process each batch of the streaming data.
Show answer and explanation
Correct answers: A, C, D
Explanation
Stream-static joins in Databricks involve combining a streaming DataFrame with a static DataFrame. To achieve this, you should broadcast the static DataFrame to optimize the join performance, use the 'join' method to specify the join condition, and enable checkpointing for fault tolerance. Watermarking is not necessary because static DataFrames do not produce late-arriving data, and 'foreachBatch' is not required for this specific scenario.
- A. Correct.
Broadcasting the static DataFrame is recommended for stream-static joins to optimize join performance and reduce shuffle operations.
- B. Incorrect.
Using a watermark is not required for stream-static joins because static DataFrames do not produce late-arriving data. Watermarks are only relevant for stream-stream joins.
- C. Correct.
The 'join' method is the correct way to perform a stream-static join. You need to specify the join condition, such as matching columns, for the join to work.
- D. Correct.
Checkpointing is essential for streaming queries to ensure fault tolerance and recovery in case of job failures.
- E. Incorrect.
The 'foreachBatch' method is not required for a simple stream-static join. This method is typically used for custom logic in batch processing.