Databricks Data Engineer Associate Question 304
Select 2In a Databricks Delta table, which type of files are compacted by the OPTIMIZE command?
- A
Small data files generated due to frequent small writes
- B
Large data files that exceed the maximum file size threshold
- C
Parquet files that are fragmented across multiple partitions
- D
Checkpoint files used for Delta Lake transaction logs
- E
Files stored in external storage systems like S3 or ADLS
Show answer and explanation
Correct answers: A, C
Explanation
The OPTIMIZE command in Databricks Delta Lake is designed to compact small data files and fragmented files within a Delta table into larger, more efficient Parquet files. This results in improved query performance and storage efficiency. It does not target large files, checkpoint files, or non-Delta table files stored in external storage directly.
- A. Correct.
Small data files generated by frequent small writes are compacted by the OPTIMIZE command to improve read performance by reducing the overhead of reading many small files.
- B. Incorrect.
Large data files that exceed the maximum size threshold are not compacted by the OPTIMIZE command since it focuses on merging small files to meet the configured target file size.
- C. Correct.
Parquet files fragmented across multiple partitions are compacted by the OPTIMIZE command to ensure better organization and efficiency in querying.
- D. Incorrect.
Checkpoint files used for Delta Lake transaction logs are not compacted by the OPTIMIZE command, as they serve a different purpose related to transaction consistency.
- E. Incorrect.
Files stored in external storage systems like S3 or ADLS are not specifically compacted by the OPTIMIZE command unless they are part of a Delta table.