Databricks Machine Learning Associate Question 503
Select 3You are working with a Spark DataFrame containing the column temperature which represents sensor readings. You suspect that the data contains outliers. You need to remove outliers from this column using either the standard deviation method or the Interquartile Range (IQR) method. Which of the following steps would correctly achieve this in PySpark?
- A
Calculate the mean and standard deviation of the
temperaturecolumn, then filter rows where the values are within three standard deviations from the mean. - B
Calculate the minimum and maximum values of the
temperaturecolumn and filter rows that fall within this range. - C
Calculate the first quartile (Q1) and third quartile (Q3) of the
temperaturecolumn, compute the IQR as Q3 - Q1, and filter rows where values are within 1.5 times the IQR below Q1 or above Q3. - D
Use the Spark DataFrame
dropna()method to remove rows with missing values, assuming outliers are represented as nulls. - E
Use the Spark SQL function
percentile_approxto calculate Q1 and Q3 for thetemperaturecolumn, derive the IQR, and filter rows where values are within the valid range based on the IQR.
Show answer and explanation
Correct answers: A, C, E
Explanation
Outliers can be removed using either the standard deviation method or the IQR method. The standard deviation method identifies values beyond a certain number of standard deviations from the mean, while the IQR method defines outliers as values outside the range determined by Q1 - 1.5 * IQR and Q3 + 1.5 * IQR. Both methods are commonly used in machine learning workflows, and leveraging Spark SQL functions like percentile_approx facilitates efficient computation in distributed environments. Filtering based on minimum/maximum values or using methods like dropna() does not specifically address outliers.
- A. Correct.
Correct. This is the standard deviation method, where values beyond three standard deviations from the mean are considered outliers. This method is commonly used for normally distributed data.
- B. Incorrect.
Incorrect. Filtering based on the minimum and maximum values of the column does not address outliers and would simply retain all valid data points.
- C. Correct.
Correct. This is the IQR method, which identifies outliers as values outside the range [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR]. This method is robust for skewed data.
- D. Incorrect.
Incorrect. While
dropna()removes rows with null values, it does not specifically address outliers, as it assumes missing values are equivalent to outliers. - E. Correct.
Correct. Using Spark SQL functions like
percentile_approxallows computation of Q1 and Q3 efficiently for large datasets, enabling the application of the IQR method to filter outliers.