Databricks Data Engineer Associate Question 89
Select 2You are working on a Databricks notebook and need to load CSV data stored in the path '/mnt/data/sales/' into a DataFrame. The directory contains multiple CSV files. Which of the following commands will correctly load the data into a DataFrame?
- A
spark.read.format('csv').load('/mnt/data/sales/')
- B
spark.read.csv('/mnt/data/sales/*.csv')
- C
spark.read.csv('/mnt/data/sales/')
- D
spark.read.format('csv').option('recursiveFileLookup', 'true').load('/mnt/data/sales/')
- E
spark.read.format('csv').load('/mnt/data/sales/file1.csv')
Show answer and explanation
Correct answers: A, C
Explanation
When reading data from a directory containing multiple CSV files in Databricks, you can either use spark.read.format('csv').load() or the shorthand spark.read.csv() with the directory path. Both methods will load all the CSV files in the directory into a single DataFrame. Additional options like recursiveFileLookup are only necessary for nested directories, and specifying individual file names or patterns is not required unless you only want specific files.
- A. Correct.
This is a correct option. Using
spark.read.format('csv').load()with a directory path loads all CSV files in the directory. - B. Incorrect.
This is incorrect. Specifying a file pattern such as '*.csv' is not required when reading from a directory in Databricks, as the API automatically loads all files in the directory when a path is provided.
- C. Correct.
This is a correct option. Using
spark.read.csv()with a directory path is a shorthand method to load all files in the directory as a DataFrame. - D. Incorrect.
This is incorrect. The option
recursiveFileLookupis unnecessary unless you are working with nested directories. The given directory path does not indicate nested directories. - E. Incorrect.
This is incorrect. This command will only load a single file ('file1.csv') from the directory, not all the files in the directory.