Databricks Data Engineer Associate Question 220
Select 3You are tasked with building a Databricks job that processes data incrementally from a source directory where new JSON files are continuously added. The processed data needs to be written to Delta Lake. Which of the following options correctly describe how you can implement incremental processing in this scenario?
- A
Use Auto Loader to read the JSON files incrementally and write the output to Delta Lake.
- B
Manually track the processed files by maintaining a separate log file and processing only untracked files.
- C
Use Delta Lake's built-in structured streaming capabilities with checkpointing to process new data incrementally.
- D
Read the entire directory using Spark's static DataFrame API and filter out already processed files based on timestamps.
- E
Configure Auto Loader with a schema inference mechanism to ensure the schema evolves with the incoming data.
Show answer and explanation
Correct answers: A, C, E
Explanation
Incremental data processing in Databricks is best achieved using tools like Auto Loader or Delta Lake's structured streaming capabilities. Auto Loader is specifically optimized for ingesting new data, while Delta Lake's structured streaming with checkpointing ensures efficient and consistent processing. Leveraging schema inference in Auto Loader can further simplify handling schema changes in incoming data.
- A. Correct.
Auto Loader is designed for efficiently ingesting new data incrementally and supports writing the output to Delta Lake.
- B. Incorrect.
Manually tracking files is not an efficient or scalable solution for incremental processing in Databricks.
- C. Correct.
Delta Lake's structured streaming with checkpointing is a common and reliable approach for incremental data processing.
- D. Incorrect.
Reading the entire directory and filtering processed files is not efficient, especially for large data sets, and is not the recommended method for incremental processing.
- E. Correct.
Auto Loader's schema inference allows handling schema evolution, which is useful for continuously added files in incremental processing scenarios.