Databricks Machine Learning Associate Question 161
Single answerYou are working with a Spark DataFrame containing a column named 'feature_value'. You need to remove outliers from this column where the values are either below 10 or above 100. Which of the following code snippets will correctly filter the DataFrame to achieve this?
- A
df.filter((col('feature_value') >= 10) & (col('feature_value') <= 100))
- B
df.filter((col('feature_value') < 10) | (col('feature_value') > 100))
- C
df.filter(col('feature_value').between(10, 100))
- D
df.filter(~(col('feature_value') < 10) & ~(col('feature_value') > 100))
Show answer and explanation
Correct answer: A
Explanation
To remove outliers from a Spark DataFrame, you need to filter rows where the column values are within the specified range (in this case, between 10 and 100). The correct approach is to use a logical AND condition, as shown in the first option. This ensures that only rows with 'feature_value' between 10 and 100 are retained, while others are removed.
- A. Correct.
This is the correct option. It uses a logical AND condition to keep rows where 'feature_value' is between 10 and 100 (inclusive), effectively removing outliers.
- B. Incorrect.
This is incorrect. It uses a logical OR condition to filter for rows where 'feature_value' is either less than 10 or greater than 100, which would retain the outliers instead of removing them.
- C. Incorrect.
This is incorrect. While
betweenis a valid method, it only works for inclusive ranges. This would retain rows between 10 and 100 but does not explicitly show how values outside this range are removed. - D. Incorrect.
This is incorrect. The use of negation (~) with separate conditions is unnecessarily complex and may lead to incorrect results or confusion. A direct AND condition is simpler and more appropriate.