Databricks Data Engineer Associate Question 87
Select 2You are a data engineer working on a Databricks notebook. You are tasked with reading sales data stored in multiple CSV files located in a directory called 'sales_data/', as well as a single file named 'summary.csv' in the same directory. Which of the following commands will correctly load the data into two distinct DataFrames?
- A
sales_df = spark.read.csv('sales_data/', header=True); summary_df = spark.read.csv('sales_data/summary.csv', header=True)
- B
sales_df = spark.read.format('csv').load('sales_data/', header=True); summary_df = spark.read.format('csv').load('sales_data/summary.csv', header=True)
- C
sales_df = spark.read.csv('sales_data/', header=True); summary_df = spark.read.csv('sales_data/summary.csv')
- D
sales_df = spark.read.format('csv').load('sales_data/'); summary_df = spark.read.format('csv').load('sales_data/summary.csv', header=True)
- E
sales_df = spark.read.csv('sales_data/', header=True); summary_df = spark.read.text('sales_data/summary.csv')
Show answer and explanation
Correct answers: A, C
Explanation
To extract data from a directory of files and a single file in Databricks, you can use spark.read.csv or spark.read.format('csv').load() methods. It is important to ensure that the 'header=True' option is used when the CSV files include a header row. The correct options demonstrate the proper syntax for reading both a directory of files and a single file into distinct DataFrames.
- A. Correct.
Correct. This command uses
spark.read.csvfor both operations with the 'header=True' option specified for reading the header row. It correctly reads the directory of files and the single file into separate DataFrames. - B. Incorrect.
Incorrect. This command improperly uses
spark.read.format('csv')and does not correctly specify the 'header=True' option for the directory of files. - C. Correct.
Correct. This command uses
spark.read.csvto read both the directory of files and the single file. The first read includes 'header=True' to process the headers, while the second read defaults to the correct behavior for a single file. - D. Incorrect.
Incorrect. This command misses the 'header=True' option while reading the directory of files, which would result in incorrect behavior for CSVs that include a header row.
- E. Incorrect.
Incorrect. This command uses
spark.read.textfor the single file, which is not appropriate for reading a CSV file.