Databricks Machine Learning Associate Question 154
Select 3You are working with a Spark DataFrame in Databricks that contains data about customer transactions. You want to compute basic summary statistics (e.g., mean, standard deviation, minimum, and maximum) for numerical columns in the DataFrame. Which of the following methods can you use to achieve this?
- A
Use the DataFrame's
describe()method - B
Use the dbutils library's
dbutils.data.summary()method - C
Use the DataFrame's
summary()method - D
Write a custom SQL query to calculate statistics using the
%sqlmagic command - E
Use the DataFrame's
stat()method and call itssummary()function
Show answer and explanation
Correct answers: A, C, D
Explanation
To compute summary statistics for numerical columns in a Spark DataFrame, you can use the built-in describe() or summary() methods. Additionally, you can leverage the %sql magic command to write SQL queries for custom statistical calculations. However, the dbutils library does not offer a method for computing summary statistics, and the stat() object does not have a summary() function. Understanding the correct methods ensures efficient computation of statistics in Databricks.
- A. Correct.
Correct: The
describe()method in Spark DataFrames computes summary statistics like count, mean, stddev, min, and max for numerical columns. - B. Incorrect.
Incorrect: The
dbutilslibrary does not have adbutils.data.summary()method. This is a fabricated option to test your knowledge. - C. Correct.
Correct: The
summary()method in Spark DataFrames is an alternative todescribe()and provides a richer set of statistics, including percentiles. - D. Correct.
Correct: Writing a custom SQL query using the
%sqlmagic command is a valid approach to compute summary statistics. However, this method requires manual query construction. - E. Incorrect.
Incorrect: While the
stat()object allows access to some statistical functions (e.g., correlation, crosstab), it does not provide asummary()function.