Databricks Data Engineer Associate Question 88
Select 2You are tasked with processing JSON data stored in Databricks. The data resides in a directory containing multiple JSON files, and you need to load all the data into a DataFrame. Which of the following code snippets would correctly load the data from the directory?
- A
spark.read.json('/path/to/directory')
- B
spark.read.json('/path/to/directory/*.json')
- C
spark.read.load('/path/to/directory', format='json')
- D
spark.read.format('json').load('/path/to/directory')
Show answer and explanation
Correct answers: A, D
Explanation
In Databricks, you can load data from a directory of files using either spark.read.json or spark.read.format('json').load. Both methods are commonly used to process JSON data and will automatically combine the contents of all JSON files in the specified directory into a single DataFrame.
- A. Correct.
This is correct because
spark.read.jsoncan directly load all the JSON files in a directory without specifying individual file names. - B. Incorrect.
This is incorrect because specifying
*.jsonis unnecessary when loading a directory, and it may lead to errors in some cases. - C. Incorrect.
This is incorrect because while
spark.read.loadis valid, you must explicitly specify that the format is JSON using.format('json'). Without it, this code will fail. - D. Correct.
This is correct because
spark.read.format('json').load('/path/to/directory')properly specifies the format and loads all JSON files in the directory.