Databricks Data Engineer Associate Question 378
Select 2You are tasked with identifying which source location in your Databricks workspace is utilizing Auto Loader. You inspect the code in several data ingestion pipelines. Which of the following code snippets indicate that Auto Loader is being used for the source location?
- A
spark.readStream.format('cloudFiles').option('cloudFiles.format', 'parquet').load('/mnt/source1')
- B
spark.read.format('parquet').load('/mnt/source2')
- C
spark.readStream.format('text').load('/mnt/source3')
- D
spark.readStream.format('cloudFiles').option('cloudFiles.format', 'json').load('/mnt/source4')
- E
spark.readStream.format('delta').load('/mnt/source5')
Show answer and explanation
Correct answers: A, D
Explanation
Auto Loader is a feature in Databricks that simplifies streaming data ingestion from cloud storage. It is identified by the use of the 'cloudFiles' format in a 'readStream' operation. Additionally, the 'cloudFiles.format' option specifies the file format being ingested. Other formats or batch operations do not use Auto Loader.
- A. Correct.
Correct: The 'cloudFiles' format in the 'readStream' operation is specific to Auto Loader. Additionally, the 'cloudFiles.format' option specifies the file format being used, making this a valid Auto Loader configuration.
- B. Incorrect.
Incorrect: This is a batch read operation using the 'read' method, not a streaming operation. Auto Loader is only used with 'readStream' for streaming ingestion.
- C. Incorrect.
Incorrect: Although this uses 'readStream', the 'text' format does not utilize Auto Loader. Auto Loader specifically requires the 'cloudFiles' format.
- D. Correct.
Correct: The 'cloudFiles' format in the 'readStream' operation indicates Auto Loader is being used. The 'cloudFiles.format' option specifies the JSON file format being ingested, making this a valid Auto Loader configuration.
- E. Incorrect.
Incorrect: This uses the Delta format with 'readStream', but it does not indicate the use of Auto Loader. Auto Loader is only applicable to the 'cloudFiles' format.