Databricks Machine Learning Associate Question 160
Single answerYou are working with a Spark DataFrame df in Databricks that contains a column value. You need to remove rows where the values in the value column are outliers, defined as being less than 10 or greater than 100. Which of the following code snippets correctly achieves this?
- A
filtered_df = df.filter((df['value'] >= 10) & (df['value'] <= 100))
- B
filtered_df = df.filter(df['value'] > 10).filter(df['value'] < 100)
- C
filtered_df = df.filter((df.value >= 10) & (df.value <= 100))
- D
filtered_df = df.filter((df['value'] > 10) | (df['value'] < 100))
Show answer and explanation
Correct answer: A
Explanation
The correct method to filter out outliers in a Spark DataFrame is to use the filter method with a logical AND (&) condition that retains rows where the value column is greater than or equal to 10 and less than or equal to 100. Using square brackets (df['column_name']) is the recommended convention in PySpark for column selection.
- A. Correct.
This is the correct code. It uses the
filtermethod with logical conditions to retain rows where thevaluecolumn is between 10 and 100 (inclusive). - B. Incorrect.
This code is incorrect because it uses two separate
filtercalls, which would exclude the boundary values (10 and 100) and does not achieve the desired logic. - C. Incorrect.
This code is incorrect because although it uses the correct syntax for accessing the column with
df.value, it does not meet the certification's best practice of using square brackets (df['column_name']) for column selection in PySpark. - D. Incorrect.
This code is incorrect because it uses the OR (
|) operator instead of the AND (&) operator, which would incorrectly include rows outside the range of 10 to 100.