Databricks Machine Learning Professional Question 155
Single answerYou are working on a machine learning pipeline in Databricks and storing prediction results in a Delta table. The prediction table is queried frequently to retrieve results for specific users based on their user IDs. You notice that the query latency is high when filtering on the user ID column. Which optimization technique can you apply to reduce the query time?
- A
Use Z-ordering on the user ID column in the Delta table.
- B
Enable Delta table versioning to optimize query performance.
- C
Repartition the Delta table based on the user ID column.
- D
Increase the number of worker nodes in the Databricks cluster.
Show answer and explanation
Correct answer: A
Explanation
Z-ordering is an effective optimization technique in Databricks for reducing query latency when filtering on specific columns. It works by co-locating related data on storage, which minimizes the amount of data read during a query. In this scenario, Z-ordering on the user ID column would improve query performance by organizing the data in a way that supports efficient filtering.
- A. Correct.
Z-ordering is a clustering technique in Delta Lake that optimizes data layout by sorting related data together in storage. Applying Z-ordering on the user ID column helps reduce the amount of data read when filtering on this column, thereby improving query performance.
- B. Incorrect.
Delta table versioning allows tracking changes to the table but does not directly improve query performance when filtering on a specific column.
- C. Incorrect.
Repartitioning can help distribute data evenly but does not specifically optimize query performance for filtering on a column like Z-ordering does.
- D. Incorrect.
Increasing the number of worker nodes may improve overall cluster performance but does not address the specific issue of optimizing query performance for filtering on the user ID column.