NCA-GENL Question 41
Select 2You are tasked with building a generative AI pipeline that uses pre-trained language models for summarizing large documents. During the preprocessing stage, you need to tokenize, lemmatize, and efficiently store vector embeddings for similarity lookups. Which combination of Python tools would best support this workflow?
- A
spaCy for tokenization and lemmatization
- B
NumPy for creating and storing vector embeddings
- C
Vector databases like FAISS or Pinecone for similarity lookups
- D
Matplotlib for visualizing embeddings
- E
TensorFlow for tokenization and lemmatization
Show answer and explanation
Correct answers: A, C
Explanation
To preprocess text for generative AI tasks, spaCy is ideal for tokenization and lemmatization, while vector databases like FAISS or Pinecone are optimized for storing and querying vector embeddings during similarity lookups. NumPy and TensorFlow, while powerful, are not the best-suited tools for these specific tasks, and Matplotlib is unrelated to the pipeline requirements.
- A. Correct.
spaCy is well-suited for tokenization and lemmatization as it is designed for efficient natural language processing tasks.
- B. Incorrect.
While NumPy can handle numerical operations, it is not specifically optimized for managing or storing vector embeddings for similarity searches.
- C. Correct.
Vector databases like FAISS or Pinecone are specifically designed for storing and performing similarity lookups on large-scale vector embeddings.
- D. Incorrect.
Matplotlib is a visualization library and is not related to processing or storing embeddings for similarity lookups.
- E. Incorrect.
TensorFlow is primarily a deep learning framework and is not typically used for tokenization and lemmatization, which are handled more efficiently by tools like spaCy.