Databricks Machine Learning Associate Question 157
Single answerYou are working on a machine learning experiment in Databricks and need to evaluate the distribution of a numeric feature in your dataset. Which of the following methods would you use to compute summary statistics such as mean, standard deviation, and quantiles for this feature?
- A
Use the
describe()method on the DataFrame containing your dataset. - B
Use the
summary()method on the DataFrame containing your dataset. - C
Use the
display()function to visualize the dataset. - D
Use the
groupBy()method followed by an aggregation function.
Show answer and explanation
Correct answer: B
Explanation
The summary() method in Databricks is specifically designed to provide detailed statistical summaries for numeric columns in a DataFrame. It is more comprehensive than the describe() method and includes statistics like quantiles, making it the correct choice for evaluating the distribution of a numeric feature.
- A. Incorrect.
The
describe()method provides basic statistics like mean, standard deviation, and count, but it does not offer a broader range of summary statistics such as quantiles. - B. Correct.
The
summary()method in Databricks provides comprehensive summary statistics, including mean, standard deviation, and quantiles, making it the appropriate choice. - C. Incorrect.
The
display()function is used for visualizing data in a tabular or graphical format but does not compute statistical summaries. - D. Incorrect.
The
groupBy()method is used for grouping data based on specific criteria, and while it can work with aggregation functions, it is not designed for computing overall summary statistics.