Databricks Machine Learning Associate Question 501
Select 3You are working with a Spark DataFrame in Databricks that contains numerical data representing sales performance metrics. You want to compute summary statistics such as mean, standard deviation, minimum, and maximum for all numerical columns in the DataFrame. Which of the following methods could you use to achieve this?
- A
Use the .summary() method on the Spark DataFrame.
- B
Use the describe() method on the Spark DataFrame.
- C
Use dbutils to generate a data profile summary.
- D
Use the .agg() method with specific aggregation functions like avg() and stddev().
- E
Use the collect() method to bring all data to the driver and calculate the statistics manually.
Show answer and explanation
Correct answers: A, C, D
Explanation
To compute summary statistics for numerical columns in a Spark DataFrame, you can use .summary() for a predefined set of statistics, dbutils data profiling for detailed summaries, or .agg() with specific aggregation functions for custom computations. Methods like describe() are limited in scope, and collect() is inefficient and not suitable for large-scale distributed data processing.
- A. Correct.
Correct. The .summary() method in Spark DataFrames computes a set of summary statistics, including mean, standard deviation, min, max, and more for numerical columns.
- B. Incorrect.
Incorrect. The describe() method only computes a limited set of statistics (count, mean, stddev, min, max) and is not as comprehensive as .summary() or dbutils-based summaries.
- C. Correct.
Correct. dbutils can be used to generate a data profile summary, which provides detailed statistics for numerical columns.
- D. Correct.
Correct. The .agg() method allows you to compute specific summary statistics by applying aggregation functions like avg() and stddev() on the DataFrame.
- E. Incorrect.
Incorrect. Using the collect() method to calculate statistics manually is inefficient and not recommended in distributed environments like Spark.