Databricks Generative AI Engineer Associate Question 224
Select 3You are working on a Databricks project where you need to create a semantic search system for a large dataset of product descriptions. You decide to use a Vector Search index to enable efficient similarity-based querying. After embedding the product descriptions into vectors using a pre-trained model, which of the following steps should you take to create and query a Vector Search index in Databricks?
- A
Store the vector embeddings in a Delta table and use Databricks’ built-in vector indexing capabilities to create the index.
- B
Use the
mlflow.vector_index.log_modelAPI to save the vector index model to MLflow. - C
Query the Vector Search index using cosine similarity or approximate nearest neighbor (ANN) techniques.
- D
Manually implement a k-nearest neighbor algorithm in Python to query the vector embeddings.
- E
Store the vector embeddings in a Spark DataFrame and use Spark SQL to compute similarity scores for queries.
Show answer and explanation
Correct answers: A, B, C
Explanation
Vector Search in Databricks involves storing vector embeddings in a Delta table, creating a vector index using tools like MLflow, and querying the index using optimized similarity techniques such as cosine similarity or approximate nearest neighbor (ANN). Using Databricks' built-in capabilities ensures efficiency and scalability, unlike manual or Spark SQL-based approaches.
- A. Correct.
Correct. In Databricks, you can store the vector embeddings in a Delta table, which serves as the foundation for creating and managing vector indexes.
- B. Correct.
Correct. The
mlflow.vector_index.log_modelAPI is used to save and log the created vector index to MLflow for efficient querying and model tracking. - C. Correct.
Correct. To query a Vector Search index, you typically use similarity techniques like cosine similarity or approximate nearest neighbor (ANN), which are supported by Databricks' vector search capabilities.
- D. Incorrect.
Incorrect. While manually implementing a k-nearest neighbor algorithm is possible, it is not recommended in Databricks as it would not leverage the built-in optimized vector search features.
- E. Incorrect.
Incorrect. Spark SQL is not designed for efficient similarity search on vector embeddings and would be computationally expensive for this use case.