Databricks Data Engineer Professional Question 114
Single answerA data engineering team is working with a large Delta Lake table containing event logs generated by IoT devices. The table includes the following columns: event_id, device_id, timestamp, and event_type. The team needs to optimize the table for archiving older data (e.g., logs older than one year) and efficiently deleting obsolete data. How should the team partition the table to best achieve these goals?
- A
Partition the table by
device_id. - B
Partition the table by
event_type. - C
Partition the table by
year(timestamp). - D
Partition the table by
device_idandevent_type.
Show answer and explanation
Correct answer: C
Explanation
Partitioning by year(timestamp) ensures that data is organized by year, which aligns with the requirement to archive or delete logs older than one year. This approach simplifies operations such as archiving or deletion of outdated data, as entire partitions for specific years can be efficiently removed without scanning the entire dataset.
- A. Incorrect.
Partitioning by
device_idmight improve query performance for device-specific data retrieval, but it does not help with archiving or deleting old data based on time (e.g., logs older than one year). - B. Incorrect.
Partitioning by
event_typecould be useful for filtering by event categories, but it does not target the requirement to archive or delete data based on time. - C. Correct.
Partitioning by
year(timestamp)groups data by year, making it easy to archive or delete data for specific years, such as removing logs older than one year. - D. Incorrect.
Partitioning by both
device_idandevent_typeincreases granularity but complicates the ability to efficiently target data based on time for archiving or deletion.