Databricks Data Engineer Professional Question 39
Single answerA data engineering team is working on a large-scale e-commerce dataset containing billions of transaction records. The dataset includes the following columns: transaction_id, customer_id, transaction_date, product_category, and amount. The team needs to optimize the dataset for query performance by partitioning it. Queries will primarily filter data by transaction_date and product_category. Which partitioning strategy should the team use?
- A
Partition by
transaction_dateonly. - B
Partition by
product_categoryonly. - C
Partition by
transaction_dateandproduct_category. - D
Partition by
customer_id.
Show answer and explanation
Correct answer: C
Explanation
Partitioning data should align with the most common query patterns to minimize the amount of data read during query execution. In this scenario, queries primarily filter data by transaction_date and product_category, so partitioning by both columns ensures efficient filtering and avoids unnecessary data scanning.
- A. Incorrect.
Partitioning by
transaction_dateonly will improve query performance for filters ontransaction_date, but it will not optimize for queries that also filter onproduct_category. - B. Incorrect.
Partitioning by
product_categoryonly will improve query performance for filters onproduct_category, but it will not optimize for queries that also filter ontransaction_date. - C. Correct.
Partitioning by both
transaction_dateandproduct_categoryallows the queries to filter data efficiently based on the most common filtering patterns, improving overall performance. - D. Incorrect.
Partitioning by
customer_idis not suitable for this scenario because the queries do not filter data based oncustomer_id.