Databricks Data Engineer Associate Question 134
Single answerYou are working with a DataFrame in Databricks that contains duplicate rows based on a combination of the 'customer_id' and 'order_date' columns. You want to deduplicate the DataFrame, keeping only the first occurrence of each duplicate based on these columns. Which of the following methods will accomplish this?
- A
Use the dropDuplicates(['customer_id', 'order_date']) method on the DataFrame.
- B
Use the distinct() method on the DataFrame without any arguments.
- C
Use the filter() method with a condition to keep only unique rows.
- D
Use the groupBy(['customer_id', 'order_date']).agg(first('*')) method.
Show answer and explanation
Correct answer: A
Explanation
To deduplicate rows based on specific columns in a DataFrame, the dropDuplicates() method is the most appropriate and efficient approach. It allows you to specify the columns to consider when identifying duplicates, ensuring only the first occurrence of each duplicate is kept.
- A. Correct.
Correct. The dropDuplicates() method is explicitly designed to deduplicate rows based on specific columns, such as 'customer_id' and 'order_date'.
- B. Incorrect.
Incorrect. The distinct() method removes all duplicate rows across the entire DataFrame, not based on specific columns.
- C. Incorrect.
Incorrect. The filter() method is used for filtering rows based on a condition, but it cannot inherently deduplicate rows.
- D. Incorrect.
Incorrect. While groupBy().agg(first('*')) can achieve deduplication, it is not an efficient or standard approach when working with large DataFrames and is not the recommended method for this task.