Databricks Machine Learning Associate Question 148
Single answerYou are working with a Spark DataFrame named sales_data that contains numerical columns revenue, profit, and cost. You want to compute summary statistics such as count, mean, standard deviation, minimum, and maximum for these columns. Which of the following commands will correctly achieve this?
- A
sales_data.summary().show()
- B
sales_data.describe().show()
- C
sales_data.summary('count', 'mean', 'stddev', 'min', 'max').show()
- D
sales_data.agg({'revenue': 'mean', 'profit': 'mean', 'cost': 'mean'}).show()
Show answer and explanation
Correct answer: A
Explanation
The .summary() method in Spark is designed specifically to compute a comprehensive set of summary statistics for all numerical columns in a DataFrame. By calling .show() on the result, you can display these statistics in a tabular format. The .describe() method is similar but more limited in scope, while .agg() is meant for custom aggregation tasks.
- A. Correct.
Correct. The
.summary()method computes summary statistics such as count, mean, standard deviation, minimum, and maximum for all numerical columns in the DataFrame. Calling.show()displays the results. - B. Incorrect.
Incorrect. The
.describe()method computes only a limited subset of summary statistics (count, mean, standard deviation, minimum, and maximum), but it lacks flexibility compared to.summary()and does not explicitly support additional statistics. - C. Incorrect.
Incorrect. While
.summary()can optionally take specific statistics as arguments, using it without arguments computes the default statistics. Adding specific arguments ('count', 'mean', etc.) would modify the behavior, but this is not necessary for the default summary statistics. - D. Incorrect.
Incorrect. The
.agg()method is used for applying aggregate functions, such as calculating the mean for specific columns, but it does not produce the full set of summary statistics for all numerical columns as.summary()does.