Databricks Data Engineer Associate Question 359
Select 2You are tasked with loading new JSON data from a cloud storage location into an existing Delta table in Databricks. The data should be appended to the table, and you want to ensure that only files that have not already been loaded are processed. Which of the following configurations should you use with the COPY INTO command?
- A
Specify the path to the cloud storage location as the source in the
COPY INTOcommand. - B
Use the
MERGEoption in theCOPY INTOcommand to avoid inserting duplicate data. - C
Enable the
cloudFiles.includeExistingFilesoption to process all files in the directory including previously loaded ones. - D
Provide a target Delta table as the destination in the
COPY INTOcommand. - E
Use the
cloudFiles.inferColumnTypesoption to automatically infer schema for the new data.
Show answer and explanation
Correct answers: A, D
Explanation
The COPY INTO command is used to efficiently load data into Delta tables, and it can automatically track files to avoid reprocessing. To achieve the task, you must specify the source path in cloud storage and the target Delta table. The other options are either not supported by COPY INTO or irrelevant to the scenario.
- A. Correct.
Correct. Specifying the cloud storage path is necessary as it tells the
COPY INTOcommand where to find the source data. - B. Incorrect.
Incorrect. The
COPY INTOcommand does not support aMERGEoption. Instead, it relies on tracking file processing to avoid duplicates. - C. Incorrect.
Incorrect. Enabling
cloudFiles.includeExistingFileswould process all files, including those that have already been loaded, which is not desired here. - D. Correct.
Correct. Providing a Delta table as the target ensures the new data is appended to the correct location.
- E. Incorrect.
Incorrect. The
cloudFiles.inferColumnTypesoption is used for schema inference in Auto Loader, not for theCOPY INTOcommand.