Databricks Data Engineer Associate Question 383
Single answerA company wants to continuously ingest JSON files containing customer transaction data from a cloud storage location into a Delta Lake table in Databricks. The files arrive in the storage location at unpredictable intervals, and the company wants to ensure minimal latency while processing only new data. Which tool or feature would be most beneficial for this use case?
- A
Using Auto Loader to automatically detect and process new files as they arrive
- B
Manually triggering a batch job in Databricks to process new files periodically
- C
Using a static schema to process all files in the storage location, regardless of arrival time
- D
Using a streaming job with manual checkpointing to track file processing
Show answer and explanation
Correct answer: A
Explanation
Auto Loader is highly beneficial in scenarios where files are ingested from cloud storage at unpredictable intervals and only new data needs to be processed. It automates the detection of new files and ensures efficient ingestion with minimal latency, making it ideal for the company’s requirements in this use case.
- A. Correct.
This is the correct option. Auto Loader is specifically designed to efficiently detect and process new files as they arrive in a cloud storage location, making it ideal for use cases involving unpredictable file arrival times.
- B. Incorrect.
Manually triggering batch jobs would not provide the minimal latency required by the company, as it involves manual intervention and periodic processing.
- C. Incorrect.
Using a static schema to process all files would result in re-processing files that have already been ingested, which is inefficient and not suited for the requirement to process only new data.
- D. Incorrect.
While a streaming job with manual checkpointing could be used, it requires additional effort to set up and manage checkpointing logic, making it less efficient than using Auto Loader for this scenario.