Databricks Machine Learning Associate Question 147
Single answerYou are working with a large dataset in Databricks and need to compute summary statistics such as mean, standard deviation, and min/max for all numerical columns in a Spark DataFrame called df. Which of the following code snippets will correctly compute these statistics?
- A
df.summary('mean', 'stddev', 'min', 'max').show() - B
df.describe('mean', 'stddev', 'min', 'max').show() - C
df.summary().show() - D
df.summary(['mean', 'stddev', 'min', 'max']).show()
Show answer and explanation
Correct answer: C
Explanation
The df.summary() method is used to compute a comprehensive set of summary statistics (e.g., mean, standard deviation, min, and max) for all numerical columns in a Spark DataFrame. By calling it without any arguments, you ensure that all predefined statistics are computed. The other options either misuse the summary method or confuse it with the describe method, which has more limited functionality.
- A. Incorrect.
This option is incorrect because while
df.summary()can compute summary statistics, you cannot specify specific metrics like 'mean', 'stddev', 'min', and 'max' as arguments tosummary. It computes a predefined set of statistics. - B. Incorrect.
This option is incorrect because the
describemethod does not accept arguments such as 'mean', 'stddev', 'min', or 'max'. It only computes a fixed set of basic statistics (count, mean, stddev, min, and max). - C. Correct.
This option is correct because calling
df.summary()with no arguments computes all predefined summary statistics (including mean, standard deviation, min, max, etc.) across all numerical columns, which is the intended functionality. - D. Incorrect.
This option is incorrect because the
summarymethod does not accept a list as an argument. Instead, it computes a fixed set of summary statistics without the need for additional inputs.