Databricks Data Engineer Associate Question 382
Select 2You are tasked with identifying which source location in a Databricks workspace is utilizing Auto Loader for streaming data ingestion. Upon inspecting the configurations of various streaming jobs, you find the following setups. Which source location(s) is/are utilizing Auto Loader?
- A
A Delta table on a Databricks-managed storage location
- B
A directory in an S3 bucket with the
cloudFilesformat specified - C
A Kafka topic configured with a structured streaming read
- D
A directory in Azure Blob Storage with the
cloudFilesformat specified - E
A table in a relational database accessed using JDBC
Show answer and explanation
Correct answers: B, D
Explanation
Auto Loader is a feature in Databricks used for incrementally and efficiently ingesting new files from cloud storage locations (e.g., AWS S3, Azure Blob Storage, or Google Cloud Storage). Auto Loader is enabled by specifying the cloudFiles format in the readStream API. In this scenario, only the source locations configured with the cloudFiles format (options 2 and 4) are utilizing Auto Loader.
- A. Incorrect.
Delta tables do not require Auto Loader for ingestion; they are already managed and optimized within Databricks.
- B. Correct.
This is correct because the
cloudFilesformat is a key feature of Auto Loader, and S3 is a supported cloud storage source for Auto Loader. - C. Incorrect.
Kafka is not a supported source for Auto Loader. It uses a different connector for real-time data ingestion.
- D. Correct.
This is correct because the
cloudFilesformat is a key feature of Auto Loader, and Azure Blob Storage is a supported cloud storage source for Auto Loader. - E. Incorrect.
JDBC is used for batch or streaming reads from relational databases, and it does not involve Auto Loader.