NCA-GENL Question 188
Select 3You are a junior developer tasked with writing a script to preprocess data for training a generative AI language model. Under the supervision of a senior team member, you are instructed to ensure the script performs tokenization, handles missing values, and converts text to lowercase. What should you include in your script to align with these instructions?
- A
Implement a function that tokenizes the text into individual words or subwords.
- B
Write a routine that replaces missing values with a default placeholder or removes them.
- C
Add functionality to convert all text data to uppercase for uniformity.
- D
Ensure the script outputs a summary of the tokenized data, such as word frequencies.
- E
Include a unit test to verify the proper handling of missing values and text transformations.
Show answer and explanation
Correct answers: A, B, E
Explanation
The correct answers are based on the instructions explicitly provided: tokenization, handling missing values, and converting text to lowercase. Additionally, including a unit test demonstrates adherence to best practices in supervised software development. The uppercase conversion and data summary options are either irrelevant or not required in this scenario.
- A. Correct.
Correct: Tokenization is a core step in preprocessing for generative AI language models and aligns with the given instructions.
- B. Correct.
Correct: Handling missing values is explicitly mentioned as a requirement in the instructions.
- C. Incorrect.
Incorrect: The instruction specifies converting text to lowercase, not uppercase, making this option irrelevant to the task.
- D. Incorrect.
Incorrect: While generating a summary of tokenized data can be helpful, it is not explicitly mentioned in the instructions as a required step.
- E. Correct.
Correct: Including a unit test ensures the script functions as intended and aligns with best practices for supervised development.