Databricks Data Engineer Professional Question 40
Single answerA data engineering team is designing a Delta Lake table to store IoT sensor data generated across thousands of devices. The table will be queried frequently for specific devices and specific time periods. The dataset is expected to grow to hundreds of terabytes. Which partitioning strategy should the team use to optimize both query performance and storage management?
- A
Partition by the device ID column only
- B
Partition by the timestamp column only (e.g., year/month/day)
- C
Partition by a combination of device ID and timestamp (e.g., year/month/day)
- D
Do not partition the table and rely on Delta Lake's file pruning capabilities
Show answer and explanation
Correct answer: C
Explanation
Partitioning strategies must be chosen based on query patterns and data growth. For IoT sensor data queried frequently by device ID and time range, partitioning by both device ID and timestamp ensures efficient data pruning during queries and avoids unnecessary data scans. This is particularly important for large datasets where both dimensions (device and time) are critical for query performance.
- A. Incorrect.
Partitioning by device ID alone is not sufficient because queries involving specific time ranges would require scanning all partitions and would not efficiently reduce the data scanned during query execution.
- B. Incorrect.
Partitioning by timestamp alone (e.g., year/month/day) could lead to small files for each device within each time partition, as there are thousands of devices generating data. This would result in inefficient storage and query performance.
- C. Correct.
Partitioning by a combination of device ID and timestamp allows for efficient pruning during queries by reducing the data scanned for both specific devices and time ranges. This balances query performance and storage management in high-growth datasets like IoT sensor data.
- D. Incorrect.
Not partitioning the table and relying on Delta Lake's file pruning capabilities would result in poor query performance for large datasets, as it would require scanning a significant amount of unnecessary data.