Databricks Data Engineer Associate Question 353
Single answerYou are working on a Databricks project where new data files are being uploaded hourly into an Azure Data Lake Storage (ADLS) directory. These files need to be ingested into a Delta Lake table for real-time analytics. The data files may arrive in small batches, and you want to ensure that only new files are processed without reprocessing files that have already been ingested. Which of the following approaches is most suitable for this scenario?
- A
Use the COPY INTO command to load the data into the Delta table, ensuring only new files are processed.
- B
Use the MERGE INTO command to ingest new data files into the Delta table and deduplicate the records.
- C
Manually track the ingested files using a metadata table and use SQL INSERT statements to insert new data into the Delta table.
- D
Use the Spark DataFrame
write.format('delta').save()method to overwrite the Delta table with new data.
Show answer and explanation
Correct answer: A
Explanation
The COPY INTO command in Databricks is optimized for incrementally loading data into Delta tables by keeping track of the files that have already been processed. In this scenario, where new data files are added hourly and only new files should be ingested, COPY INTO is the most suitable choice. It avoids the need for manual file tracking and ensures efficient data ingestion.
- A. Correct.
This is correct. The COPY INTO command is designed for loading data into Delta tables incrementally by tracking and ingesting only new files. It is ideal for this scenario where hourly files are added, and reprocessing of previously ingested files should be avoided.
- B. Incorrect.
This is incorrect. The MERGE INTO command is primarily used for upserts (insert and update operations) based on specific conditions, not for incrementally ingesting new files.
- C. Incorrect.
This is incorrect. Manually tracking files in a metadata table and using SQL INSERT statements can be error-prone and inefficient compared to using COPY INTO, which automates file tracking.
- D. Incorrect.
This is incorrect. Overwriting the Delta table with
write.format('delta').save()would replace all existing data rather than incrementally adding new data.