Databricks Machine Learning Associate Question 506
Select 2You are working with a Spark DataFrame containing a column named 'measurement' that has extreme values (outliers). You need to remove these outliers using either the standard deviation method or the interquartile range (IQR) method. Which of the following approaches correctly identify and remove outliers from the 'measurement' column?
- A
Calculate the mean and standard deviation of the 'measurement' column, filter rows where the values are within 3 standard deviations from the mean.
- B
Calculate the mean and standard deviation of the 'measurement' column, filter rows where the values are outside 3 standard deviations from the mean.
- C
Calculate the first quartile (Q1) and third quartile (Q3) of the 'measurement' column, and filter rows where values are within the range [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR].
- D
Calculate the first quartile (Q1) and third quartile (Q3) of the 'measurement' column, and filter rows where values are outside the range [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR].
- E
Use a correlation matrix to identify outlier values in the 'measurement' column and remove those rows.
Show answer and explanation
Correct answers: A, C
Explanation
Outliers can be removed using the standard deviation method or the IQR method. The standard deviation method filters rows where values are within a certain number of standard deviations (commonly 3) from the mean, while the IQR method filters rows where values are within the range defined by [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR]. Understanding these methods is essential for preprocessing data in Spark DataFrames.
- A. Correct.
Correct. Removing outliers using the standard deviation method involves filtering rows where the values are within a certain number of standard deviations (commonly 3) from the mean.
- B. Incorrect.
Incorrect. Filtering rows where values are outside 3 standard deviations is the inverse of the correct approach. This would incorrectly keep the outliers and remove most normal data points.
- C. Correct.
Correct. Removing outliers using the IQR method involves filtering rows where values fall within the range [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR]. This is a standard approach for identifying and removing outliers.
- D. Incorrect.
Incorrect. Filtering rows where values are outside the IQR range is the inverse of the correct approach. This would incorrectly keep the outliers and remove most normal data points.
- E. Incorrect.
Incorrect. A correlation matrix measures the relationship between variables, not outlier detection in a single column. This method is not applicable for removing outliers.