Databricks Machine Learning Associate Question 497
Select 4You are working on summarizing a Spark DataFrame named sales_df in Databricks. You want to compute descriptive statistics such as mean, standard deviation, and count for all numeric columns in the DataFrame. Which of the following methods or workflows will achieve this?
- A
Use
sales_df.summary()to calculate summary statistics. - B
Use
dbutils.data.summarize(sales_df)to generate a summary of the DataFrame. - C
Use
sales_df.describe()to calculate summary statistics. - D
Use
sales_df.statistics()to calculate summary statistics. - E
Use SQL to query the DataFrame by registering it as a TempView and using SQL aggregate functions like AVG and COUNT.
Show answer and explanation
Correct answers: A, B, C, E
Explanation
The .summary() method allows you to compute a wide range of descriptive statistics for a Spark DataFrame, while .describe() provides basic summary statistics. In Databricks, dbutils.data.summarize can also be used to get a detailed summary of the DataFrame, including visualizations. Additionally, using SQL queries on a registered TempView is a standard method for computing aggregate statistics in Databricks. However, .statistics() is not a valid method, so it is incorrect.
- A. Correct.
sales_df.summary()is a valid method in PySpark to calculate summary statistics such as mean, standard deviation, and count for all numeric columns in the DataFrame. - B. Correct.
dbutils.data.summarize(sales_df)is a valid method in Databricks for generating summary statistics and visualizations of a DataFrame. - C. Correct.
sales_df.describe()is a PySpark method that computes basic statistics such as mean, count, and standard deviation, but its scope is more limited compared to.summary(). - D. Incorrect.
sales_df.statistics()is not a valid PySpark or Databricks method for computing summary statistics, so this option is incorrect. - E. Correct.
Registering the DataFrame as a TempView and using SQL aggregate functions like AVG and COUNT is a valid alternative approach to calculate summary statistics in Databricks.