Databricks Data Engineer Professional Question 154
Select 3You are tasked with implementing a streaming pipeline in Databricks that consumes data from multiple bronze tables containing raw, unstructured logs. The goal is to ensure the best practices for processing these streams to avoid duplication, ensure schema enforcement, and optimize downstream performance. Which of the following actions should you take?
- A
Use checkpointing to maintain state and ensure exactly-once processing.
- B
Apply schema inference on each read operation to dynamically handle schema changes in the bronze tables.
- C
Leverage watermarking to handle late-arriving data and avoid unbounded state.
- D
Read from the bronze tables using Delta Lake's 'readStream' API and write the output to a silver table in Delta format.
- E
Directly write streaming output from the bronze tables into a downstream reporting dashboard for real-time insights.
Show answer and explanation
Correct answers: A, C, D
Explanation
When streaming data from multiplex bronze tables, it's important to follow best practices such as using checkpointing for fault tolerance, leveraging watermarking to handle late-arriving data, and transforming data into a silver table for optimized downstream use. Avoid relying on schema inference for production workloads and avoid skipping intermediate transformations that ensure data quality and consistency.
- A. Correct.
Checkpointing is essential for maintaining state in streaming operations, ensuring fault tolerance, and supporting exactly-once processing. This is a best practice for streaming from bronze tables.
- B. Incorrect.
While schema inference may seem convenient, it's not a best practice in production environments, especially for bronze tables with evolving schemas. Explicit schema definition ensures consistency and avoids runtime issues.
- C. Correct.
Watermarking is a best practice for handling late-arriving data and preventing unbounded state in streaming queries. This is especially important when processing raw logs with potential delays.
- D. Correct.
Using Delta Lake's 'readStream' API to read from bronze tables and writing to a silver table ensures data is transformed, cleaned, and optimized for downstream consumption. This aligns with the medallion architecture best practices.
- E. Incorrect.
Directly writing streaming output to a reporting dashboard is not recommended because it bypasses critical transformations and optimizations typically performed in the silver table. This can lead to data quality and performance issues.