Databricks Generative AI Engineer Associate Question 248
Single answerYou are fine-tuning a large language model (LLM) on a customer support dataset to improve response accuracy while meeting strict latency requirements. During training, you observe that excessive token generation is causing a performance bottleneck. Which masking technique can you use to enforce guardrails and meet the latency objective?
- A
Apply attention masking to limit the model's focus to relevant tokens.
- B
Use padding masking to ensure uniform sequence lengths during training.
- C
Implement top-k sampling masking to restrict token selection to the most probable k tokens.
- D
Enable causal masking to prevent the model from attending to future tokens.
Show answer and explanation
Correct answer: C
Explanation
Top-k sampling masking is particularly effective in scenarios where performance objectives such as latency need to be met. By limiting the token selection process to the most probable k tokens, it reduces the computational cost during inference and ensures that the model generates responses more quickly without sacrificing too much quality. Other masking techniques like attention masking, padding masking, and causal masking serve different purposes and do not directly address latency concerns caused by excessive token generation.
- A. Incorrect.
Attention masking is used to focus on specific portions of the input, but it does not directly address performance bottlenecks caused by excessive token generation.
- B. Incorrect.
Padding masking ensures uniform sequence lengths by ignoring pad tokens during training, but it does not help in reducing latency or token generation issues.
- C. Correct.
Top-k sampling masking restricts the token selection process to the k most likely tokens, reducing the computational overhead and ensuring faster token generation, which directly addresses the latency issue.
- D. Incorrect.
Causal masking ensures that the model only attends to past tokens, which is essential for autoregressive generation but does not mitigate the issue of excessive token generation affecting latency.