Databricks Data Engineer Associate Question 379
Select 2A data engineering team is working on a Databricks project and is using Auto Loader to ingest data from various sources. The team needs to identify which source location is configured to utilize Auto Loader. Below are the possible configurations for their data sources. Which source location(s) is correctly set up to utilize Auto Loader?
- A
A cloud storage location (e.g., s3://bucket-name/data) with a specified schema and the 'cloudFiles' format used in the DataFrame reader
- B
A local file system directory (e.g., /tmp/data) read using the 'csv' format in the DataFrame reader
- C
A Delta Lake table read using the 'delta' format in the DataFrame reader
- D
A cloud storage location (e.g., azure://container-name/data) with the 'cloudFiles' format and options configured for schema inference
- E
A Kafka topic read using the 'kafka' format in the DataFrame reader
Show answer and explanation
Correct answers: A, D
Explanation
Auto Loader is a feature in Databricks designed to efficiently ingest data from cloud storage locations like AWS S3, Azure Blob Storage, and Google Cloud Storage. To utilize Auto Loader, the 'cloudFiles' format must be specified in the DataFrame reader, along with appropriate options like schema inference. Auto Loader does not support local file systems, Delta Lake tables, or Kafka topics as source locations.
- A. Correct.
Correct. Auto Loader supports ingesting data from cloud storage locations (e.g., S3, Azure Blob Storage, Google Cloud Storage) when the 'cloudFiles' format is used in the DataFrame reader.
- B. Incorrect.
Incorrect. Auto Loader does not support local file systems. It is specifically designed for scalable, cloud-based data ingestion.
- C. Incorrect.
Incorrect. While Delta Lake is a powerful feature in Databricks, it is not a source location for Auto Loader. Auto Loader is used to ingest raw data, not read Delta tables.
- D. Correct.
Correct. Auto Loader can ingest data from supported cloud storage locations such as Azure Blob Storage, provided the 'cloudFiles' format is used with any necessary options configured (e.g., schema inference).
- E. Incorrect.
Incorrect. Auto Loader does not support Kafka as a source. Kafka is a streaming platform, and Auto Loader is designed for file-based data ingestion from cloud storage.