Databricks Machine Learning Associate Question 163
Single answerYou are working with a Spark DataFrame containing numerical data in a column named 'value'. You want to remove outliers where the values are above 100 or below 10. Which of the following Spark operations would achieve this?
- A
df.filter((df['value'] >= 10) & (df['value'] <= 100))
- B
df.where((df['value'] > 100) | (df['value'] < 10))
- C
df.filter((df['value'] > 10) & (df['value'] < 100))
- D
df.withColumn('value', when((df['value'] >= 10) & (df['value'] <= 100), df['value']).otherwise(None))
Show answer and explanation
Correct answer: A
Explanation
To remove outliers from a Spark DataFrame, you can use the filter() operation to retain only the rows where the 'value' column falls within the specified range. The correct answer, df.filter((df['value'] >= 10) & (df['value'] <= 100)), ensures that only rows with 'value' between 10 and 100 (inclusive) are retained, effectively removing the outliers.
- A. Correct.
This is the correct option. The filter operation ensures that only rows with 'value' between 10 and 100 (inclusive) are retained in the DataFrame.
- B. Incorrect.
This is incorrect. The where clause here filters rows where 'value' is either greater than 100 or less than 10, which keeps the outliers instead of removing them.
- C. Incorrect.
This is incorrect. The filter operation here removes values equal to 10 or 100, which is not aligned with the requirement to include values between 10 and 100 (inclusive).
- D. Incorrect.
This is incorrect. While this approach creates a new column and sets outlier values to None, it does not remove the rows containing outliers from the DataFrame.