Databricks Machine Learning Associate Question 145
Single answerYou are working with a Spark DataFrame df containing numeric data. You want to compute summary statistics such as mean, standard deviation, minimum, and maximum for all numeric columns in the DataFrame. Which of the following code snippets will correctly achieve this?
- A
df.summary().show()
- B
df.describe().show()
- C
df.summary('mean', 'stddev', 'min', 'max').show()
- D
df.stats().summary().show()
Show answer and explanation
Correct answer: A
Explanation
The .summary() method in Spark DataFrames computes extensive summary statistics for numeric columns, including mean, standard deviation, min, max, and other measures. It is more comprehensive than .describe() and does not require additional parameters to calculate the default statistics.
- A. Correct.
Correct. The
.summary()method computes summary statistics for all numeric columns in the DataFrame. Using.show()will display the results. - B. Incorrect.
Incorrect. The
.describe()method computes basic statistics (count, mean, stddev, min, max) but does not provide the flexibility to calculate additional statistics like.summary(). - C. Incorrect.
Incorrect. While
.summary()can take specific statistics as arguments, the provided statistics ('mean', 'stddev', 'min', 'max') are already included by default in.summary(). The correct way to compute this is simply usingdf.summary().show(). - D. Incorrect.
Incorrect. There is no method
df.stats()available in PySpark. This is an invalid code snippet.