Databricks Machine Learning Associate Question 504
Select 2You are working with a Spark DataFrame containing a column named 'sales'. You suspect that the column contains outliers and want to remove them. Which of the following approaches can be used to filter out outliers based on standard deviation or interquartile range (IQR) in PySpark?
- A
Calculate the mean and standard deviation of the 'sales' column, then filter rows where values fall outside 3 standard deviations from the mean.
- B
Compute the first (Q1) and third quartiles (Q3) of the 'sales' column, calculate the IQR as Q3 - Q1, and filter rows where values fall outside the range [Q1 - 1.5IQR, Q3 + 1.5IQR].
- C
Use the describe() method on the DataFrame to directly identify and remove outliers based on standard deviation.
- D
Write a custom UDF to flag and remove outliers using the IQR method.
- E
Use Spark’s built-in 'removeOutliers()' function to automatically filter outliers based on either standard deviation or IQR.
Show answer and explanation
Correct answers: A, B
Explanation
Removing outliers in PySpark requires explicit calculations using either the standard deviation or IQR method. Both approaches involve computing summary statistics (e.g., mean, standard deviation, quantiles) and filtering rows based on a threshold. The describe() method and UDFs are not efficient or sufficient for this task, and no built-in function like 'removeOutliers()' exists in Spark.
- A. Correct.
This is correct. Calculating the mean and standard deviation, then filtering rows using a threshold (e.g., 3 standard deviations from the mean) is a valid and common approach to removing outliers in PySpark.
- B. Correct.
This is correct. The IQR method is a standard approach for identifying outliers by calculating Q1 and Q3, deriving the IQR, and filtering rows outside the specified range. This can be implemented using PySpark functions like approxQuantile().
- C. Incorrect.
This is incorrect. The describe() method provides summary statistics (e.g., mean, standard deviation, min, max) but does not directly support outlier removal.
- D. Incorrect.
This is incorrect. While UDFs can be used for row-level operations in PySpark, they are inefficient for this task given that PySpark provides built-in functions to compute summary statistics like mean, standard deviation, and quantiles.
- E. Incorrect.
This is incorrect. Spark does not provide a built-in 'removeOutliers()' function. Outlier removal must be explicitly implemented using standard deviation or IQR calculations.