Databricks Data Engineer Associate Question 385
Single answerYou are working on a data pipeline that ingests real-time log files generated by various applications into a Delta Lake table in Databricks. The log files are continuously written to a cloud storage location in an unstructured format. You need a solution that can efficiently detect new files in the storage location, handle schema evolution, and process the data incrementally without requiring manual intervention. Which Databricks feature is most appropriate for this use case?
- A
Auto Loader
- B
Structured Streaming with a static file source
- C
Batch processing using COPY INTO
- D
Delta Live Tables
Show answer and explanation
Correct answer: A
Explanation
Auto Loader is beneficial in scenarios where data files are continuously written to a cloud storage location, and you need to process them incrementally. It is optimized for detecting new files, handling schema evolution, and reducing manual intervention, making it the ideal choice for this use case.
- A. Correct.
Auto Loader is specifically designed for incrementally and efficiently ingesting data from cloud storage. It supports schema evolution, can automatically detect new files, and is optimized for streaming or incremental data processing.
- B. Incorrect.
Structured Streaming with a static file source is not suitable for detecting new files as it works with a predefined set of files and does not automatically monitor for new ones.
- C. Incorrect.
Batch processing using COPY INTO is not optimized for continuously detecting new files or for handling schema evolution automatically. It is more suited for ad-hoc batch loading.
- D. Incorrect.
Delta Live Tables is a powerful feature for building ETL pipelines but is not specifically optimized for detecting new files in a cloud storage location like Auto Loader.