Databricks Data Engineer Professional Question 109
Single answerYou are working on a Databricks project where your team needs to maintain a large dataset of customer transaction records. The dataset is updated daily, and regulatory requirements mandate that all transactions older than 7 years must be archived or deleted efficiently. How should you partition the data to simplify this process?
- A
Partition the data by customer ID to group transactions by individual customers.
- B
Partition the data by transaction type to group similar types of transactions together.
- C
Partition the data by transaction year to enable efficient filtering of older transactions.
- D
Do not use partitioning and instead rely on full table scans to filter and delete old transactions.
Show answer and explanation
Correct answer: C
Explanation
Partitioning data by transaction year is the correct approach because it aligns with the requirement to manage data retention based on age. By organizing the data into year-based partitions, you can quickly identify and process only the partitions that meet the 7-year threshold, improving efficiency and scalability.
- A. Incorrect.
Partitioning by customer ID does not simplify the process of archiving or deleting old transactions since the data is grouped by customer rather than by time.
- B. Incorrect.
Partitioning by transaction type groups similar transactions, but it does not help with filtering or deleting data based on age, which is the primary requirement here.
- C. Correct.
Partitioning by transaction year allows you to efficiently filter out or delete transactions older than 7 years by targeting specific year-based partitions, avoiding a full table scan.
- D. Incorrect.
Not using partitioning would result in inefficient full table scans, making the deletion or archiving process slow and resource-intensive.