Databricks Machine Learning Professional Question 153
Single answerYou are working on a machine learning project that generates predictions for millions of rows in a Delta table. Data scientists frequently query the table to analyze predictions for a specific subset of records based on a column called prediction_date. These queries often take a long time to execute. How can you optimize the table to improve query performance in this scenario?
- A
Use Z-Ordering on the
prediction_datecolumn to physically organize data for faster query performance. - B
Enable Delta Cache to store the entire table in memory for faster access.
- C
Partition the table by the
prediction_datecolumn to divide the data across multiple files. - D
Convert the Delta table into a Parquet table to reduce storage overhead.
Show answer and explanation
Correct answer: A
Explanation
Z-Ordering is a powerful optimization technique in Delta Lake that organizes data files based on the values of specific columns, such as prediction_date. This reduces the number of files read during queries that filter on these columns, significantly improving query performance for large datasets.
- A. Correct.
Z-Ordering is an optimization technique that co-locates related information in the same data files. Applying Z-Ordering on the
prediction_datecolumn ensures that queries filtering on this column will read fewer files, improving performance. - B. Incorrect.
Delta Cache can improve performance for frequently accessed data, but it is not as effective as Z-Ordering for optimizing queries filtered on a specific column like
prediction_date. - C. Incorrect.
Partitioning by
prediction_datecould help, but it may lead to too many small files when the number of unique dates is high. This can degrade performance instead of improving it. - D. Incorrect.
Converting the Delta table to a Parquet table would lose the ACID transactional benefits of Delta Lake and does not inherently improve query performance in this scenario.