Databricks Data Engineer Associate Question 130
Single answerYou are working on a Databricks notebook, and you are tasked with deduplicating rows in a DataFrame named orders_df based on the order_id column. However, for rows with the same order_id, you want to keep the row with the most recent order_date. Which of the following code snippets achieves this?
- A
orders_df.orderBy('order_id', 'order_date').dropDuplicates(['order_id'])
- B
orders_df.orderBy('order_id', 'order_date').distinct()
- C
orders_df.orderBy('order_id', F.desc('order_date')).dropDuplicates(['order_id'])
- D
orders_df.dropDuplicates(['order_id']).orderBy(F.desc('order_date'))
Show answer and explanation
Correct answer: C
Explanation
To deduplicate a DataFrame based on a specific column (order_id) while keeping the most recent row (based on order_date), the DataFrame should first be ordered by the column to prioritize (descending order_date in this case). After ordering, dropDuplicates can be applied to remove duplicate rows based on the specified column while retaining the desired row.
- A. Incorrect.
This code does not ensure that the most recent
order_dateis kept. TheorderByis in ascending order (default), so the oldest date will be retained during deduplication. - B. Incorrect.
This code does not address deduplication based on
order_id. Thedistinct()function only removes exact duplicate rows and does not consider specific columns likeorder_id. - C. Correct.
This code correctly orders the DataFrame by
order_idand descendingorder_date, ensuring the most recent date is at the top for eachorder_id. The subsequentdropDuplicates(['order_id'])ensures deduplication while keeping the most recent row. - D. Incorrect.
This code applies deduplication first using
dropDuplicates(['order_id'])without guaranteeing the most recentorder_dateis kept, asorderByis applied after deduplication, which won't change the already deduplicated result.