Databricks Data Engineer Associate Question 381
Single answerYou are tasked with analyzing a Databricks workspace to identify which source location is utilizing Auto Loader for streaming data ingestion. After inspecting the notebook configurations, you find the following code snippets. Which one indicates the use of Auto Loader?
- A
spark.readStream.format('csv').load('/mnt/data/source')
- B
spark.readStream.format('cloudFiles').option('cloudFiles.format', 'json').load('/mnt/data/source')
- C
spark.read.format('csv').load('/mnt/data/source')
- D
spark.read.format('cloudFiles').option('cloudFiles.format', 'parquet').load('/mnt/data/source')
Show answer and explanation
Correct answer: B
Explanation
Auto Loader is Databricks' feature for incrementally and efficiently processing new files as they arrive in cloud storage. It requires the 'cloudFiles' format to be specified in the 'spark.readStream' operation. Among the provided options, only the second snippet uses both 'spark.readStream' and 'cloudFiles', correctly indicating the use of Auto Loader.
- A. Incorrect.
This code uses 'spark.readStream' but does not use the 'cloudFiles' format, which is required for Auto Loader. It is a standard streaming read operation and does not leverage Auto Loader.
- B. Correct.
This code uses 'spark.readStream' with the 'cloudFiles' format and specifies the data format as 'json'. The use of 'cloudFiles' indicates Auto Loader is being utilized for streaming data ingestion.
- C. Incorrect.
This code uses 'spark.read' for batch processing, not 'spark.readStream', and does not use the 'cloudFiles' format. It is a batch operation, not a streaming operation with Auto Loader.
- D. Incorrect.
This code uses 'spark.read' instead of 'spark.readStream', and although it includes the 'cloudFiles' format, it is not a streaming operation. Auto Loader is only applicable for streaming data ingestion.