Databricks Machine Learning Associate Question 151
Single answerYou are working with a Spark DataFrame in Databricks and want to compute summary statistics, such as mean, standard deviation, min, and max, for a column named 'age'. You decide to use Databricks utilities (dbutils) to perform the task. Which of the following steps will allow you to achieve this?
- A
Use the
describe()function on the DataFrame and display the results usingshow(). - B
Call
dbutils.data.summarize()to compute and display the summary statistics. - C
Use the
summary()function on the DataFrame and collect the results. - D
Call
dbutils.fs.stats()on the DataFrame to summarize the 'age' column.
Show answer and explanation
Correct answer: A
Explanation
To compute summary statistics for a Spark DataFrame, you can use the describe() function, which provides basic statistics like count, mean, stddev, min, and max for numeric columns. The show() method is then used to display these results. While the summary() function is also an option, it provides a more detailed set of metrics and is not the best fit for the specific requirements described in the question.
- A. Correct.
Correct. The
describe()function in Spark provides summary statistics (count, mean, stddev, min, max) for numeric columns. Displaying the results withshow()will allow you to see the computed statistics. - B. Incorrect.
Incorrect.
dbutils.data.summarize()is not a valid Databricks utility for computing summary statistics. This is a fictional method. - C. Incorrect.
Incorrect. While the
summary()function can be used to compute various statistics, it provides a more detailed summary thandescribe(). However, the question explicitly asks about basic summary statistics, makingdescribe()the more accurate choice. - D. Incorrect.
Incorrect.
dbutils.fs.stats()is used to retrieve file system metadata, such as file size and modification time, and is unrelated to DataFrame computations.