Databricks Data Engineer Associate Question 86
Single answerYou are tasked with ingesting JSON files stored in a Databricks workspace. The JSON files are located in a directory, and you need to load all the data into a DataFrame for processing. Which of the following Spark commands should you use to correctly extract data from the directory?
- A
spark.read.json('/mnt/data-directory/')
- B
spark.read.format('json').load('/mnt/data-directory/*.json')
- C
spark.read.format('json').load('/mnt/data-directory/')
- D
spark.read.json('/mnt/data-directory/file.json')
Show answer and explanation
Correct answer: C
Explanation
To extract data from a directory containing multiple JSON files, you should use spark.read.format('json').load('/mnt/data-directory/'). This method explicitly specifies the JSON format and loads all files in the directory. Using a wildcard pattern like *.json is not necessary, as Spark automatically reads all the JSON files in the specified directory.
- A. Incorrect.
This is incorrect because while it can read JSON files from a directory, the
spark.read.jsonfunction automatically uses the directory path without requiring wildcard patterns like*.json. The wildcard is unnecessary when reading from a directory. - B. Incorrect.
This is incorrect because using the
*.jsonwildcard is redundant and not required when reading all files in a directory. Spark can infer the file format and read all JSON files directly from the directory path. - C. Correct.
This is correct because
spark.read.format('json').load('/mnt/data-directory/')is the proper way to read all JSON files from a directory using the DataFrame API. It specifies the format explicitly and loads all files from the directory. - D. Incorrect.
This is incorrect because it specifies a single file path, not a directory. If you want to read from a directory, you need to provide the directory path, not the path of an individual file.