Databricks Data Engineer Professional Question 110
Select 2You are working with a large dataset of IoT sensor readings stored in a Delta table. The dataset is partitioned by a device_type column. The business requires you to archive older records for a specific device type and delete records for another device type. Which approach leverages the partitioning to efficiently meet these requirements?
- A
Use the Delta table's
VACUUMcommand to delete all records for a specificdevice_type. - B
Filter the Delta table by
device_typeand write the older records for archiving to cloud storage. - C
Delete records for the specific
device_typeusing theDELETEcommand with a filter on thedevice_typecolumn. - D
Repartition the Delta table dynamically before performing the archiving and deletion tasks.
- E
Use the
OPTIMIZEcommand on the Delta table to automatically remove records for the specificdevice_type.
Show answer and explanation
Correct answers: B, C
Explanation
Partitioning the Delta table by device_type allows efficient filtering and targeting of specific partitions for both archiving and deletion tasks. By filtering and writing records to an archive location, and using the DELETE command to remove data for a specific device_type, you can avoid scanning unrelated partitions and improve performance. The other options either misuse Delta commands or add unnecessary overhead.
- A. Incorrect.
The
VACUUMcommand is used for cleaning up old files that are no longer referenced by the Delta transaction log. It does not delete active records or filter data based on conditions likedevice_type. - B. Correct.
This is a correct approach. By filtering the Delta table based on the
device_typecolumn, you can identify older records for a specific device type and write them to an archive location. The partitioning bydevice_typeimproves efficiency by scanning only the relevant partitions. - C. Correct.
This is a correct approach. Since the Delta table is partitioned by
device_type, theDELETEcommand can efficiently target and remove records for the specificdevice_typewithout scanning unnecessary partitions. - D. Incorrect.
Repartitioning the Delta table dynamically is unnecessary in this scenario, as the table is already partitioned by
device_type. Repartitioning would add overhead without improving efficiency for this task. - E. Incorrect.
The
OPTIMIZEcommand is used to compact small files in Delta tables to improve query performance. It does not delete records or archive data for specific partitions.