Databricks Data Engineer Associate Question 350
Select 2A Data Engineer executes a COPY INTO statement to load data into a Delta table. However, upon running the statement multiple times with the same source files, no duplicate data is added to the target table. What could explain this behavior?
- A
The COPY INTO statement uses the FILES parameter to specify an already processed file.
- B
Delta Lake automatically deduplicates data when using COPY INTO statements.
- C
COPY INTO only appends new data if the source files have not been previously loaded into the target table.
- D
The target Delta table has a UNIQUE constraint applied to prevent duplicates.
- E
The COPY INTO statement is configured with a FILEFORMAT option that filters duplicate rows.
Show answer and explanation
Correct answers: A, C
Explanation
COPY INTO tracks previously processed files in the Delta Lake transaction log. If a file has already been loaded into the target table, it is skipped in subsequent executions, preventing duplicate data. This behavior is especially relevant when using the FILES parameter to specify particular files or when re-running COPY INTO with the same source location.
- A. Correct.
Correct. If the FILES parameter is used, COPY INTO will only process the specified files, and Delta Lake tracks which files have already been processed to prevent duplication.
- B. Incorrect.
Incorrect. Delta Lake does not automatically deduplicate data in COPY INTO statements. Deduplication needs to be explicitly handled, such as by using DISTINCT or MERGE operations.
- C. Correct.
Correct. COPY INTO tracks the source file paths in its transaction log and skips files that have already been processed, ensuring no duplicate data is added.
- D. Incorrect.
Incorrect. Delta Lake does not enforce UNIQUE constraints natively. Duplicate prevention in COPY INTO is managed via file tracking, not constraints.
- E. Incorrect.
Incorrect. The FILEFORMAT option specifies the file type (e.g., CSV, JSON) for parsing but does not filter duplicate rows during ingestion.