Databricks Machine Learning Associate Question 162
Single answerYou are working with a Spark DataFrame named df that contains a column feature with numerical values. You want to remove rows where the values of feature are less than 10 or greater than 100. Which of the following code snippets will correctly filter out these outliers?
- A
filtered_df = df.filter((df['feature'] >= 10) & (df['feature'] <= 100))
- B
filtered_df = df.where((df['feature'] >= 10) & (df['feature'] <= 100))
- C
filtered_df = df.filter((df['feature'] > 10) & (df['feature'] < 100))
- D
filtered_df = df.where((df['feature'] > 10) | (df['feature'] < 100))
Show answer and explanation
Correct answer: A
Explanation
To remove outliers, you must correctly apply a filtering condition that retains rows where values are within the specified range. The filter method is commonly used in PySpark for this purpose. The correct solution ensures the range is inclusive by using >= and <= operators.
- A. Correct.
Correct. The
filtermethod is used to retain rows where the condition is true. Here, it ensuresfeaturevalues are between 10 and 100 (inclusive). - B. Incorrect.
While
whereis functionally similar tofilter, this option is incorrect because the exam question specifically asks about using thefilterfunction. - C. Incorrect.
Incorrect. The usage of
>and<excludes the boundary values (10 and 100), which does not meet the requirement of including them. - D. Incorrect.
Incorrect. The condition uses a logical OR (
|), which would retain rows wherefeatureis either greater than 10 or less than 100, leading to incorrect filtering.