Databricks Machine Learning Associate Question 502
Select 3You are working on a Spark DataFrame named sales_data containing a column revenue. You suspect there are outliers in the revenue column. Which of the following approaches could you use in Databricks to remove these outliers effectively?
- A
Filter rows where the
revenuecolumn values fall outside 3 standard deviations from the mean. - B
Filter rows where the
revenuecolumn values fall outside the interquartile range (IQR) by more than 1.5 times the IQR. - C
Use the
fillna()method to replace all null values in therevenuecolumn with the median value. - D
Drop rows with outlier values in the
revenuecolumn by setting a threshold based on domain knowledge. - E
Apply a clustering algorithm like K-Means to group similar rows and consider outliers as data points in smaller clusters.
Show answer and explanation
Correct answers: A, B, D
Explanation
Outlier removal in Spark DataFrames can be achieved using statistical methods like standard deviation or IQR-based filtering, which are computationally efficient and widely used in machine learning workflows. Alternatively, domain knowledge can be applied to set thresholds for filtering data. Methods like fillna() address missing values and are not relevant for outliers, while clustering algorithms are not directly intended for outlier removal.
- A. Correct.
This is a correct approach. Removing outliers based on standard deviation is a common statistical method to handle extreme values.
- B. Correct.
This is a correct approach. Filtering outliers based on the IQR is widely used in data preprocessing for skewed data distributions.
- C. Incorrect.
This is incorrect because
fillna()is used to handle missing values, not outliers. - D. Correct.
This is a correct approach. Setting a threshold based on domain expertise is a practical way to identify and remove outliers specific to a dataset.
- E. Incorrect.
This is incorrect because clustering techniques like K-Means are not specifically designed for outlier removal and require additional steps to identify outliers.