Databricks Generative AI Engineer Associate Question 45
Single answerYou are designing a pipeline to preprocess a set of legal documents for fine-tuning a generative AI model. The documents contain long paragraphs, tables, and bullet points. The model you are using has a maximum token limit of 4096 tokens for input. Which chunking strategy would be most appropriate to ensure no information is lost while adhering to the model's token constraints?
- A
Split the document by fixed-size character counts to ensure each chunk fits within the token limit.
- B
Chunk the document based on semantic boundaries, such as paragraphs or sections, while ensuring each chunk stays within the token limit.
- C
Combine all tables and bullet points into a single chunk to preserve their structure, regardless of token size.
- D
Randomly split the document into smaller pieces to balance chunk sizes across the document.
Show answer and explanation
Correct answer: B
Explanation
When applying a chunking strategy for documents, it is important to preserve the semantic structure and meaning of the content while respecting the token constraints of the model. Splitting by semantic boundaries such as paragraphs or sections ensures that the chunks are meaningful and coherent, making them suitable for input to the generative AI model. This approach minimizes information loss and enhances the quality of the model's outputs.
- A. Incorrect.
Splitting by fixed-size character counts can lead to information being cut off mid-sentence or mid-paragraph, which might compromise the semantic meaning of the content.
- B. Correct.
Chunking based on semantic boundaries, such as paragraphs or sections, ensures the integrity of the content while adhering to the model's token limit, making it the most appropriate strategy.
- C. Incorrect.
Combining all tables and bullet points into a single chunk may exceed the token limit and does not preserve the overall document structure.
- D. Incorrect.
Randomly splitting the document disrupts the logical flow of information, which can negatively impact model performance and fine-tuning.