Databricks Data Engineer Professional Question 112
Single answerA data engineering team is tasked with managing a Delta Lake table containing transaction data for a financial application. The table is expected to grow by millions of records daily, and regulatory requirements mandate that records older than 7 years must be archived or deleted. How can the team use proper partitioning to simplify the archiving or deletion of old data?
- A
Partition the Delta Lake table by transaction date, allowing efficient filtering and targeted operations on older records.
- B
Partition the Delta Lake table by customer ID to group transactions by customer and simplify archiving.
- C
Use a single directory for all data without partitioning to reduce complexity and improve query performance.
- D
Partition the Delta Lake table by the record size to store smaller and larger records separately for faster processing.
Show answer and explanation
Correct answer: A
Explanation
Partitioning the Delta Lake table by transaction date allows the team to efficiently filter and target specific date ranges for deletion or archiving. This approach minimizes the amount of data scanned during operations, improving performance and ensuring compliance with regulatory requirements. Partitioning by other attributes, such as customer ID or record size, does not address the need for date-based data management and can complicate the process.
- A. Correct.
Partitioning by transaction date enables efficient filtering of data for deletion or archiving based on date ranges, meeting the requirement to manage records older than 7 years.
- B. Incorrect.
Partitioning by customer ID does not simplify the deletion or archiving of old data since the operation is based on transaction dates, not customer-specific information.
- C. Incorrect.
Using a single directory without partitioning leads to performance challenges when filtering or managing large datasets, especially when handling date-specific operations.
- D. Incorrect.
Partitioning by record size is unrelated to the problem and does not address date-based operations for deletion or archiving.