Databricks Data Engineer Associate Question 146
Single answerYou are tasked with validating that a value is NOT present in the 'status' column of a Delta table named 'transactions'. The column should not contain the value 'failed'. Which of the following options correctly implements this validation in a PySpark DataFrame?
- A
df.filter(df['status'] != 'failed').count() == df.count()
- B
df.filter(df['status'] == 'failed').count() == 0
- C
df.filter(df['status'].isNotNull()).count() == df.count()
- D
df.filter(df['status'] != 'failed').show()
Show answer and explanation
Correct answer: B
Explanation
To validate that a specific value is not present in a column, we need to ensure that the count of rows containing that value is zero. The correct answer achieves this by filtering for rows with 'status' equal to 'failed' and confirming their count is zero.
- A. Incorrect.
This option checks that all rows where the 'status' is not 'failed' match the total row count. However, this doesn't confirm that 'failed' is completely absent, as there could still be rows with 'status' equal to 'failed'.
- B. Correct.
This option correctly validates that the count of rows with 'status' equal to 'failed' is zero, meaning the value 'failed' is not present in the column.
- C. Incorrect.
This option checks that all rows in the 'status' column are not null, which is unrelated to validating the absence of the specific value 'failed'.
- D. Incorrect.
This option displays the rows where the 'status' is not 'failed', but it does not validate whether 'failed' is absent from the column.