Databricks Machine Learning Associate Question 499
Single answerYou are working with a Spark DataFrame in Databricks that contains information about sales transactions, including columns for 'product_id', 'quantity', and 'price'. You want to compute summary statistics such as mean, standard deviation, min, and max for all numeric columns in the DataFrame. Which of the following approaches would allow you to achieve this?
- A
Use the .summary() method on the Spark DataFrame.
- B
Use the .describe() method on the Spark DataFrame.
- C
Use dbutils.data.summarize() on the Spark DataFrame.
- D
Use the .summary('count', 'mean') method to compute only count and mean.
Show answer and explanation
Correct answer: A
Explanation
The .summary() method is the most appropriate way to compute a wide range of summary statistics (e.g., mean, stddev, min, max) for all numeric columns in a Spark DataFrame. While other methods like .describe() or dbutils.data.summarize() provide some functionality for summarizing data, they either lack the full range of statistics or are designed for different use cases, such as interactive exploration.
- A. Correct.
Correct: The .summary() method computes summary statistics (e.g., mean, stddev, min, max) for all numeric columns in a Spark DataFrame.
- B. Incorrect.
Incorrect: The .describe() method provides basic summary statistics (count, mean, stddev, min, max), but its output is less flexible and does not include all the statistics provided by .summary().
- C. Incorrect.
Incorrect: dbutils.data.summarize() provides an interactive summary of the data in the Databricks Notebook UI but is not used for programmatically computing summary statistics in a Spark DataFrame.
- D. Incorrect.
Incorrect: While .summary('count', 'mean') can compute specific statistics, the question asks for all summary statistics, which requires calling .summary() without arguments.