Databricks Machine Learning Associate Question 153
Select 3You are working with a Spark DataFrame in Databricks containing customer purchase data. You want to compute summary statistics for this DataFrame, such as mean, standard deviation, and count for numeric columns. Which of the following methods can you use to achieve this?
- A
Use the describe() method on the Spark DataFrame.
- B
Use the dbutils.data.summary() function.
- C
Use the summary() method on the Spark DataFrame.
- D
Use the groupBy() method followed by agg() with statistical functions.
- E
Use the pandas_profiling.ProfileReport() directly on the Spark DataFrame.
Show answer and explanation
Correct answers: A, C, D
Explanation
To compute summary statistics on a Spark DataFrame in Databricks, you can use the describe() or summary() methods for predefined statistics, or use groupBy() with agg() for custom aggregations. The dbutils module is not used for this purpose, and pandas_profiling is not directly applicable to Spark DataFrames without conversion.
- A. Correct.
The describe() method on a Spark DataFrame computes basic summary statistics like count, mean, stddev, min, and max for numeric columns. This is a valid option.
- B. Incorrect.
The dbutils.data module does not provide a summary() or similar function. dbutils is primarily used for utilities like file system operations, not for computing statistics.
- C. Correct.
The summary() method on a Spark DataFrame provides a more detailed set of summary statistics compared to describe(), including quantiles. This is a valid option.
- D. Correct.
Using the groupBy() method followed by agg() with statistical functions allows for custom aggregations and computation of summary statistics. This is a valid approach.
- E. Incorrect.
pandas_profiling.ProfileReport() is a pandas-specific library and cannot be used directly on a Spark DataFrame. You would need to convert the Spark DataFrame to a pandas DataFrame first, which is not mentioned in the context.