Databricks Machine Learning Associate Question 150
Single answerYou are working on a Databricks notebook and need to compute summary statistics (like mean, min, max, and standard deviation) for a Spark DataFrame called sales_data. To ensure the computation runs efficiently and you can inspect the results interactively, which approach should you use?
- A
Use the
df.describe()method on thesales_dataDataFrame and call.show()to display the results. - B
Use the
dbutils.fs.summary()function to compute summary statistics for the DataFrame. - C
Convert the
sales_dataDataFrame to Pandas and use the Pandasdescribe()method. - D
Use the
sales_data.stat.summary()method to compute summary statistics and write the output to a table usingdbutils.fs.put().
Show answer and explanation
Correct answer: A
Explanation
The describe() method on a Spark DataFrame is specifically designed to compute summary statistics efficiently in distributed environments. It avoids the memory limitations of Pandas and does not require additional file system operations like dbutils.fs.put(). Using .show() further allows users to interactively inspect the results in a Databricks notebook environment. The other options are either incorrect or inefficient for this use case.
- A. Correct.
df.describe()is a Spark DataFrame method that efficiently computes summary statistics for numeric columns. The.show()command allows you to view the results interactively in your notebook. - B. Incorrect.
dbutils.fs.summary()does not exist. Thedbutilslibrary is primarily used for file system operations and is not meant for computing statistics on Spark DataFrames. - C. Incorrect.
While converting a Spark DataFrame to Pandas and using
describe()is possible, this approach is not efficient or scalable for large datasets because Pandas operates in memory and may fail for big data. - D. Incorrect.
sales_data.stat.summary()is an incorrect method name. While Spark does providestatfunctions for some statistical operations, the correct method for summary statistics isdescribe(). Writing the results to a table usingdbutils.fs.put()is not necessary for this task.