Chunk size is a trade-off. Very small chunks carry too little surrounding context, so a passage that only makes sense together with its neighbours becomes ambiguous on its own. Very large chunks carry so much text that the specific answer is buried among unrelated sentences, diluting what the model reads. A workable chunk size keeps one coherent idea together without dragging in the rest of the document.
How AI Answers Questions About Documents It Never Learned