Databricks Data Engineer Associate Question 90
Single answerYou are working on a Databricks notebook and need to extract data from a directory containing multiple CSV files. The directory is located in an S3 bucket at 's3://your-bucket/data/'. How can you read all the files in the directory into a single DataFrame?
- A
Use the
spark.read.csv('s3://your-bucket/data/')command. - B
Use the
spark.read.csv('s3://your-bucket/data/*.csv')command. - C
Write a loop to read all files individually and use
unionto combine them into a single DataFrame. - D
Use the
spark.read.format('csv').load('s3://your-bucket/data/')command.
Show answer and explanation
Correct answer: A
Explanation
To read all files in a directory of a specific format (e.g., CSV), you can directly use spark.read.csv() with the directory path. Spark automatically processes all files within the directory that match the specified format. This is the simplest and most efficient way to load data from a directory.
- A. Correct.
This is the correct way to read all files in a directory into a single DataFrame in Databricks. The
spark.read.csv()function automatically reads all files in the specified directory. - B. Incorrect.
Although this syntax may seem valid, it is unnecessary to specify the wildcard
*.csvwhen reading an entire directory using Spark. Spark automatically reads all files in the directory with the specified format. - C. Incorrect.
While this approach could work, it is inefficient and not recommended because Spark is capable of reading all files in a directory into a single DataFrame without requiring a loop or manual union operations.
- D. Incorrect.
Although this uses a valid Spark method,
load()without specifying the format specifically would not work correctly for CSV files. You would need to add.option('header', 'true')and other relevant options to ensure proper parsing.