Databricks Data Engineer Associate Question 357
Single answerYou are working on a Databricks project where you need to incrementally load new data files from an external cloud storage location into a Delta table. The files are delivered daily in a specific directory, and you want to ensure that only new files are added to the Delta table without reprocessing files that have already been loaded. Which approach should you choose?
- A
Use the COPY INTO command to load the new files into the Delta table.
- B
Use the MERGE INTO command to merge the new files into the Delta table.
- C
Use the INSERT INTO command to insert all files into the Delta table.
- D
Use the LOAD DATA command to load the files into the Delta table.
Show answer and explanation
Correct answer: A
Explanation
The COPY INTO command is the correct choice for this scenario because it is explicitly designed for incrementally loading data into a Delta table from cloud storage. It keeps track of the files that have already been ingested, ensuring that only new files are processed, which avoids unnecessary reprocessing and improves efficiency.
- A. Correct.
The COPY INTO command is specifically designed for incrementally loading new data files into a Delta table. It tracks loaded files and ensures that only new files are processed.
- B. Incorrect.
The MERGE INTO command is used for upserts (insertions and updates) based on matching conditions between source and target datasets. While powerful, it is not optimal for simply loading new files incrementally.
- C. Incorrect.
The INSERT INTO command inserts data into a table but does not handle incremental loading or avoid reprocessing of already-loaded files.
- D. Incorrect.
The LOAD DATA command is not supported in Databricks for Delta Lake operations, making it an invalid choice for this scenario.