Databricks Machine Learning Professional Question 9
Select 2You are working on a Databricks notebook and need to create a Delta table to store the output of a machine learning pipeline. After running the pipeline, you want to read the Delta table to validate the results. Which of the following steps correctly demonstrate how to write and read data from a Delta table using PySpark in Databricks?
- A
Use the
.write.format('delta').save('/path/to/delta_table')method to write the DataFrame to a Delta table and the.read.format('delta').load('/path/to/delta_table')method to read it. - B
Use the
.write.delta('/path/to/delta_table')method to write the DataFrame to a Delta table and the.read.delta('/path/to/delta_table')method to read it. - C
Use the
.saveAsTable('delta_table')method to save the Delta table in a managed location and usespark.table('delta_table')to read it. - D
Use the
.write.option('format', 'delta').save('/path/to/delta_table')method to write the DataFrame to a Delta table and the.read.option('format', 'delta').load('/path/to/delta_table')method to read it. - E
Use the
.write.format('parquet').save('/path/to/delta_table')method to write the DataFrame to a Delta table and the.read.format('delta').load('/path/to/delta_table')method to read it.
Show answer and explanation
Correct answers: A, C
Explanation
To correctly work with Delta tables in Databricks, you need to either use .write.format('delta') and .read.format('delta') for file-based storage or use .saveAsTable() and spark.table() for managed tables. Options 1 and 3 demonstrate these two correct approaches. Other options either use incorrect syntax or incompatible formats.
- A. Correct.
Correct: This is the standard way to write a DataFrame to a Delta table and read it back by specifying the 'delta' format using
.write.format()and.read.format(). - B. Incorrect.
Incorrect: There is no
.write.delta()or.read.delta()method in PySpark. The correct format should be specified using.write.format('delta')and.read.format('delta'). - C. Correct.
Correct: This is an alternative approach where you can save a Delta table as a managed table using
.saveAsTable()and then read it back usingspark.table(). - D. Incorrect.
Incorrect: While
.option('format', 'delta')might seem valid, the correct syntax for Delta tables uses.format('delta')in both writing and reading operations. - E. Incorrect.
Incorrect: Writing a Delta table using
.format('parquet')will not create a Delta table but rather a Parquet file. This makes it incompatible with.read.format('delta').