Databricks Machine Learning Associate Question 500
Select 2You are working with a Spark DataFrame named sales_data in Databricks that contains information about sales transactions, including columns for 'product_id', 'quantity', and 'price'. You want to compute summary statistics such as mean, standard deviation, minimum, and maximum for all numeric columns in the DataFrame. Which of the following methods can you use to achieve this?
- A
sales_data.summary().show()
- B
dbutils.data.summaries(sales_data)
- C
sales_data.describe().show()
- D
sales_data.stat.summary().show()
- E
sales_data.select('quantity', 'price').summary().show()
Show answer and explanation
Correct answers: A, E
Explanation
The .summary() method in Spark DataFrames is designed to compute detailed summary statistics like mean, standard deviation, minimum, and maximum for numeric columns. It can be used on the entire DataFrame or specific selected columns. The .describe() method is more limited in scope, and other listed methods like .stat.summary() and dbutils.data.summaries() do not exist in this context.
- A. Correct.
Correct: The
.summary()method provides descriptive statistics such as mean, standard deviation, minimum, and maximum for all numeric columns in the DataFrame, and.show()is used to display the results. - B. Incorrect.
Incorrect: There is no method
dbutils.data.summaries()available in Databricks or Spark for computing summary statistics. - C. Incorrect.
Incorrect: The
.describe()method provides limited summary statistics (only count, mean, standard deviation, min, and max) but does not include extended metrics like the.summary()method. While valid in some contexts, it does not fully meet the requirements of the scenario. - D. Incorrect.
Incorrect: The method
.stat.summary()is not a valid function in Spark or Databricks. The correct method is simply.summary(). - E. Correct.
Correct: The
.summary()method can be applied to specific columns (e.g., 'quantity' and 'price') using the.select()function, and.show()displays the resulting summary statistics.