AIF-C01 Question 107
Select 2You are deploying a text generation model using Amazon SageMaker and want to control the randomness and creativity of the model's responses. Additionally, you need to ensure that the responses are concise and do not exceed a specific number of tokens. Which inference parameters should you adjust to achieve this?
- A
Temperature
- B
Max tokens
- C
Top-k sampling
- D
Learning rate
- E
Batch size
Show answer and explanation
Correct answers: A, B
Explanation
To control the randomness and creativity of a model's responses, you adjust the 'Temperature' parameter. To limit the output length and ensure concise responses, you use the 'Max tokens' parameter. Other parameters like learning rate and batch size are not relevant to inference behavior, while Top-k sampling could influence randomness but is less central to this specific scenario.
- A. Correct.
Temperature controls the randomness of the model's responses. A lower temperature results in more deterministic outputs, while a higher temperature allows for more creative and varied responses. This parameter directly impacts the randomness of the model's predictions.
- B. Correct.
Max tokens determines the maximum length of the output generated by the model. This is crucial for ensuring that the responses remain concise and within a predefined token limit.
- C. Incorrect.
Top-k sampling is another inference parameter that controls randomness by limiting the model's choices to the top-k most probable tokens. However, it is not directly as relevant as temperature or max tokens in this specific scenario.
- D. Incorrect.
Learning rate is a parameter used during model training to control the step size during optimization. It does not affect inference behavior and is irrelevant in this context.
- E. Incorrect.
Batch size is a parameter related to the number of input samples processed together during inference or training. It does not directly influence the randomness or length of the model's responses.