Databricks Data Engineer Professional Question 47
Select 2You are tasked with designing a data pipeline to process clickstream data collected from a web application. The data is stored in a Delta table and updated every few minutes. The business requires both historical trend analysis and near real-time insights. Which approach should you use to efficiently process the data while minimizing resource usage?
- A
Use batch processing to periodically process the entire Delta table for historical insights and incremental processing to handle new data for real-time reports.
- B
Use only batch processing to process the entire Delta table periodically for both historical and near real-time insights.
- C
Use a streaming query with structured streaming to continuously process new data into an aggregation table for real-time insights while maintaining a separate batch process for historical analysis.
- D
Use only a streaming query with structured streaming to process both historical and new data for all insights.
- E
Implement a caching mechanism to store intermediate results from the Delta table for faster query execution and avoid incremental processing.
Show answer and explanation
Correct answers: A, C
Explanation
The optimal approach to meet the business requirements of both historical trend analysis and near real-time insights is to use a combination of batch processing for historical data and incremental processing via structured streaming for real-time updates. This ensures efficient resource utilization and timely insights without overcomplicating the pipeline or incurring unnecessary compute costs.
- A. Correct.
Correct: Combining batch processing for historical insights with incremental processing for real-time data is an efficient and practical solution that balances resource usage and performance.
- B. Incorrect.
Incorrect: Solely relying on batch processing is inefficient for near real-time insights, as it processes the entire dataset repeatedly, leading to unnecessary resource consumption.
- C. Correct.
Correct: Using structured streaming for incremental updates ensures that new data is processed in near real-time, while batch processing handles historical data separately, aligning with the business requirements.
- D. Incorrect.
Incorrect: Relying solely on streaming for both historical and new data processing is unnecessarily complex and resource-intensive, especially for historical trend analysis.
- E. Incorrect.
Incorrect: While caching can improve query performance, it does not replace the need for incremental processing or appropriately designed batch workflows.