Databricks Machine Learning Associate Question 505
Select 2You have a Spark DataFrame containing a column named 'price' with numerical values. You suspect there are outliers in the 'price' column and want to remove them using either the standard deviation or the Interquartile Range (IQR) method. Which of the following code snippets correctly removes outliers from the DataFrame?
- A
df = df.filter((col('price') > mean_price - 3 * stddev_price) & (col('price') < mean_price + 3 * stddev_price))
- B
df = df.filter((col('price') > q1 - 1.5 * iqr) & (col('price') < q3 + 1.5 * iqr))
- C
df = df.filter((col('price') > mean_price - stddev_price) & (col('price') < mean_price + stddev_price))
- D
df = df.filter((col('price') > q1 - 3 * iqr) & (col('price') < q3 + 3 * iqr))
- E
df = df = df.filter((col('price') > q3 - 1.5 * iqr) & (col('price') < q1 + 1.5 * iqr))
Show answer and explanation
Correct answers: A, B
Explanation
Outliers can be removed using standard deviation or the IQR method. The standard deviation method filters values within three standard deviations from the mean, while the IQR method filters values outside 1.5 times the IQR below Q1 or above Q3. Both approaches are valid and commonly used for outlier detection.
- A. Correct.
Correct: This is the standard deviation method, which filters values within three standard deviations from the mean. It is a valid approach to remove outliers.
- B. Correct.
Correct: This is the IQR-based method, which removes values outside 1.5 times the IQR below Q1 or above Q3. It is commonly used for outlier detection.
- C. Incorrect.
Incorrect: While this uses the standard deviation, it only filters within one standard deviation from the mean, which may remove too many valid data points and is not typically used for outlier removal.
- D. Incorrect.
Incorrect: While this uses the IQR, multiplying it by 3 instead of 1.5 is not a standard approach and may allow too many outliers to remain in the data.
- E. Incorrect.
Incorrect: This reverses Q1 and Q3 in the IQR calculation, which is incorrect and would not properly identify outliers.