Databricks Data Engineer Professional Question 113
Single answerYou are working with a large Delta table in Databricks that stores logs from IoT devices. The table is partitioned by a device_type column and a log_date column (in the format yyyy-MM-dd). The business has requested a monthly archival of logs older than one year and the ability to delete logs for specific device types upon request. Which of the following partitioning practices would best support efficient archiving and deletion of data?
- A
Partition by both
device_typeandlog_dateto enable efficient filtering and deletion based on these columns. - B
Partition by only the
log_datecolumn to simplify the partition structure and improve query performance. - C
Partition by a hash of the
device_typecolumn to evenly distribute data across partitions. - D
Do not use partitioning and rely solely on file pruning for data deletion and archival.
Show answer and explanation
Correct answer: A
Explanation
Partitioning by both device_type and log_date aligns with the business requirements for efficient data archival and deletion. This approach allows the system to prune partitions based on the specified log_date and device_type, reducing the amount of data scanned. The other options either do not address both requirements or result in reduced efficiency.
- A. Correct.
Partitioning by both
device_typeandlog_dateallows for targeted filtering, deletion, and archival of data based on these specific columns, which aligns with the requirements in the scenario. - B. Incorrect.
Partitioning by only
log_datewould make it difficult to efficiently delete data for specificdevice_typevalues, as all device types would be grouped together within the same date partitions. - C. Incorrect.
Partitioning by a hash of the
device_typecolumn does not directly aid in the archival or deletion of data based onlog_dateor specificdevice_typevalues, making it an unsuitable choice for this scenario. - D. Incorrect.
Not using partitioning would result in poor performance for archival or deletion operations, as the entire dataset would need to be scanned to locate the relevant data.