Databricks Data Engineer Associate Question 355
Single answerYou are working as a data engineer for a company that ingests daily transactional data from an external vendor into a Delta table in Databricks. The files are delivered as partitioned Parquet files into an Azure Data Lake Storage container. You need to efficiently load these files into your Delta table, ensuring that the process handles schema evolution and avoids reloading files that were already ingested. Which approach would be the most appropriate?
- A
Use the COPY INTO command to load the files into the Delta table.
- B
Manually write a Spark job to read the files and write them into the Delta table.
- C
Use the MERGE INTO command to load the files into the Delta table.
- D
Use the INSERT INTO command to load the files into the Delta table.
Show answer and explanation
Correct answer: A
Explanation
The COPY INTO command is the most appropriate choice for this scenario because it is specifically designed to load data efficiently from external sources into Delta tables. COPY INTO automatically handles schema evolution, tracks already ingested files to avoid duplicates, and works well with partitioned data. This makes it the ideal solution for ingesting daily transactional data delivered as partitioned Parquet files.
- A. Correct.
This is the correct answer. The COPY INTO command is specifically designed for efficiently loading data into Delta tables from external sources. It supports schema evolution, avoids reloading files by tracking already loaded files, and is optimized for loading partitioned data.
- B. Incorrect.
While writing a custom Spark job could achieve the same goal, it would require significantly more effort to handle schema evolution, deduplication, and file tracking manually. COPY INTO provides these features out-of-the-box.
- C. Incorrect.
MERGE INTO is used for upserts and merges rather than bulk loading new data into a Delta table. It is not the appropriate choice for this scenario.
- D. Incorrect.
INSERT INTO is not suitable for efficiently loading large amounts of partitioned data or handling schema evolution. It also does not track already ingested files, which can lead to duplicate data.