NCA-GENL Question 167
Select 2You are building a natural language processing (NLP) pipeline for a document classification task. The pipeline requires tokenization, word embeddings, and similarity search for querying a vector database. Which Python libraries would you use to accomplish these tasks efficiently?
- A
spaCy for tokenization, NumPy for vector operations, and FAISS for similarity search
- B
spaCy for tokenization, pandas for similarity search, and NumPy for embeddings
- C
spaCy for tokenization, Gensim for word embeddings, and FAISS for similarity search
- D
NumPy for tokenization, spaCy for similarity search, and Gensim for embeddings
- E
FAISS for embeddings, NumPy for tokenization, and pandas for similarity search
Show answer and explanation
Correct answers: A, C
Explanation
For the described NLP pipeline, spaCy is a popular choice for tokenization due to its speed and accuracy. Gensim provides pre-trained word embeddings and tools for working with word vectors, while FAISS is a highly efficient library for similarity search in vector databases. NumPy can be used for vector operations but not embeddings or tokenization directly, and pandas is not a library designed for similarity search. Therefore, the correct combinations are Options 1 and 3.
- A. Correct.
spaCy is well-suited for tokenization, NumPy handles vector operations efficiently, and FAISS is a robust library for fast similarity search in vector databases. This combination is efficient for the described NLP task.
- B. Incorrect.
While spaCy is suitable for tokenization, pandas is not designed for similarity search, and NumPy does not provide pre-trained word embeddings directly. This makes this option unsuitable for the pipeline.
- C. Correct.
spaCy is effective for tokenization, Gensim provides pre-trained word embeddings, and FAISS excels in vector similarity search. This option aligns well with the requirements of the task.
- D. Incorrect.
NumPy is not designed for tokenization, and using spaCy for similarity search is incorrect because it is primarily an NLP library. Gensim is suitable for embeddings, but the overall combination is not efficient for the described pipeline.
- E. Incorrect.
FAISS is not used for generating word embeddings, NumPy is not a tokenization library, and pandas is not efficient for similarity search tasks. This makes this option invalid for the given use case.