Databricks Data Engineer Associate Question 341
Single answerA data engineering team is tasked with maintaining a Delta Lake table containing customer transaction data. The table may receive duplicate records due to upstream data issues. The team wants to ensure that only the latest records for each transaction are retained when writing new data into the table. Which command should the team use to deduplicate the data in this scenario?
- A
MERGE INTO
- B
INSERT OVERWRITE
- C
UPDATE
- D
DELETE
Show answer and explanation
Correct answer: A
Explanation
The MERGE INTO command is the most appropriate choice for deduplicating data in Delta Lake tables. It allows you to combine new incoming data with existing data by matching records on a key condition and applying deduplication logic. Other commands like INSERT OVERWRITE, UPDATE, and DELETE do not provide the required functionality for deduplication.
- A. Correct.
The MERGE INTO command is specifically designed for scenarios where records need to be updated, inserted, or deduplicated based on matching conditions. It allows you to compare incoming data with existing data and apply specific actions (e.g., deduplication).
- B. Incorrect.
INSERT OVERWRITE replaces the entire data in a table or partition but does not support conditional logic for deduplication. This makes it unsuitable for this scenario.
- C. Incorrect.
UPDATE modifies existing records but does not handle deduplication or insert new records, making it insufficient for the given requirements.
- D. Incorrect.
DELETE removes records from a table but does not support inserting or updating records, so it cannot deduplicate the data.