Databricks Machine Learning Associate Question 498
Select 3You are working with a Spark DataFrame named sales_data in Databricks, which contains information about monthly sales, including columns month, region, and revenue. Your task is to compute summary statistics for the revenue column, such as mean, standard deviation, and min/max values. Which of the following approaches would correctly compute these statistics?
- A
Use
sales_data.summary('mean', 'stddev', 'min', 'max').show()to compute summary statistics for all numeric columns, includingrevenue. - B
Use
dbutils.data.summarize(sales_data)to display an interactive summary of the DataFrame, including statistics forrevenue. - C
Use
sales_data.describe('revenue').show()to compute and display basic summary statistics like count, mean, stddev, min, and max for therevenuecolumn. - D
Use
sales_data.select('revenue').summary().show()to compute and display extended summary statistics specifically for therevenuecolumn. - E
Use
sales_data.groupBy('region').summary().show()to compute summary statistics for therevenuecolumn grouped byregion.
Show answer and explanation
Correct answers: A, B, D
Explanation
To compute summary statistics for a Spark DataFrame, you can use the .summary() method for extended statistics or .describe() for basic statistics. Additionally, dbutils.data.summarize() can be used for an interactive exploration of the DataFrame. Grouping operations are not directly supported within .summary() or .describe() and require explicit aggregation functions.
- A. Correct.
This is a correct approach. The
.summary()method computes extended summary statistics for all numeric columns in the DataFrame, includingrevenue. By specifying statistics like 'mean' or 'stddev', you can narrow down the results. - B. Correct.
This is a correct approach.
dbutils.data.summarize()provides an interactive summary of the DataFrame, including statistics for numeric columns likerevenue. However, this is more for exploration than programmatic use. - C. Incorrect.
This is incorrect because the
.describe()method provides only basic summary statistics (count, mean, stddev, min, max) and cannot compute extended statistics like percentiles. Additionally, it is less flexible than.summary(). - D. Correct.
This is a correct approach. Using
.summary()on a specific column (selected with.select()) computes extended summary statistics like mean, stddev, min, max, and percentiles for that column. - E. Incorrect.
This is incorrect because
.summary()does not support grouping by a column. If you need grouped statistics, you must use.groupBy()with aggregation functions explicitly.