Databricks Data Engineer Associate Question 117
Single answerYou are working on a Databricks notebook and have a DataFrame named sales_data with the following schema: id INT, product STRING, price DOUBLE. Some rows in the price column contain NULL values. You want to count the number of rows in the DataFrame where the price column has non-NULL values. Which of the following commands will produce the correct result?
- A
sales_data.count()
- B
sales_data.select('price').count()
- C
sales_data.filter(sales_data['price'].isNotNull()).count()
- D
sales_data.filter(sales_data['price'].isNull()).count()
Show answer and explanation
Correct answer: C
Explanation
The count() function in PySpark/DataFrame operations does not inherently skip NULL values unless the DataFrame is filtered to exclude NULLs beforehand. The correct way to count rows with non-NULL values in the price column is to use the filter method with isNotNull() to exclude NULL rows before applying the count() function.
- A. Incorrect.
This command counts the total number of rows in the DataFrame, including rows where the
pricecolumn is NULL. Therefore, it does not meet the requirement. - B. Incorrect.
This command counts all rows in the
pricecolumn, but it does not exclude rows where the value is NULL. Hence, it does not meet the requirement. - C. Correct.
This command filters the rows where the
pricecolumn is NOT NULL and then counts those rows. This correctly provides the count of rows with non-NULL values in thepricecolumn. - D. Incorrect.
This command filters the rows where the
pricecolumn is NULL and then counts those rows. However, it does the opposite of what is required.