NCA-GENL Question 38
Select 2You are building a text classification pipeline using Python. To preprocess the text data, you need to tokenize the input, remove stop words, and transform the text into numerical vectors for downstream machine learning tasks. Which combination of Python libraries would be most effective for this task?
- A
spaCy for tokenization, stop word removal, and NumPy for vector transformations
- B
NumPy for tokenization and stop word removal, and spaCy for vector transformations
- C
spaCy for tokenization and stop word removal, and a vector database like FAISS for efficient vector storage
- D
NumPy for all preprocessing tasks including tokenization, stop word removal, and vectorization
- E
Using spaCy for tokenization and stop word removal combined with NumPy for numerical vector transformations
Show answer and explanation
Correct answers: A, E
Explanation
The best combination for text preprocessing in this scenario is to use spaCy for tokenization and stop word removal, as it is designed for natural language processing tasks. NumPy complements this by providing efficient numerical operations to transform text into vectorized representations. Other options incorrectly assume NumPy can handle text preprocessing or misuse FAISS, which is optimized for vector storage and similarity search, not preprocessing.
- A. Correct.
Correct: spaCy provides robust methods for tokenization and stop word removal, while NumPy is well-suited for numerical transformations of text data into vectors.
- B. Incorrect.
Incorrect: NumPy does not handle tokenization or stop word removal, it is primarily a numerical computation library.
- C. Incorrect.
Partially correct: spaCy is effective for tokenization and stop word removal, but FAISS is a vector database optimized for similarity search, not for preprocessing tasks like vector transformations.
- D. Incorrect.
Incorrect: NumPy alone cannot handle tokenization, stop word removal, or text-specific preprocessing tasks; it is used for numerical operations.
- E. Correct.
Correct: Combining spaCy's text preprocessing capabilities with NumPy's numerical operations is an effective approach for this scenario.