Databricks Data Engineer Associate Question 119
Single answerYou are working with a Spark DataFrame sales_data that contains the following columns: id (unique identifier), product (product name), and price (numeric value, which can contain NULLs). You want to count the total number of rows in the DataFrame, including those where the price column is NULL. Which of the following approaches will correctly count all rows, including those with NULL values in any column?
- A
Use
sales_data.count() - B
Use
sales_data.select('price').count() - C
Use
sales_data.filter('price IS NOT NULL').count() - D
Use
sales_data.agg({'price': 'count'}).collect()[0][0]
Show answer and explanation
Correct answer: A
Explanation
The DataFrame.count() method counts all the rows in the DataFrame regardless of whether any column contains NULL values. This is different from aggregations like count on a specific column, which skip NULL values in that column. Therefore, to count all rows in the DataFrame, including those with NULL values in any column, the correct method is sales_data.count().
- A. Correct.
sales_data.count()counts all rows in the DataFrame, including those with NULL values in any column. This is the correct method to count the total number of rows in the DataFrame. - B. Incorrect.
sales_data.select('price').count()counts all rows in thepricecolumn, but since it is still counting rows, it will include NULL values. However, this is an unnecessary operation when you simply want to count all rows of the DataFrame. - C. Incorrect.
sales_data.filter('price IS NOT NULL').count()counts only the rows where thepricecolumn is not NULL, which excludes rows with NULL values inprice. This is not what is required in this scenario. - D. Incorrect.
sales_data.agg({'price': 'count'}).collect()[0][0]counts non-NULL values in thepricecolumn only. This method skips rows wherepriceis NULL, so it does not return the total count of all rows.