asta-nguyen8/8/2026, 6:03:04 AM

split_to_token_limit adjusted end backward to a UTF-8 char boundary
but advanced start by a fixed step (max_bytes - overlap_bytes).
When end was adjusted back, bytes between end and start+step were
skipped — dropping code points at chunk boundaries.

Fix: start = end.saturating_sub(overlap_bytes) instead of start += step.
This ensures every byte from end onward is in the next chunk.

Reproduced: 20 CJK chars (60 bytes), max_tokens=2, overlap=0
  Before: 7 chunks, 42 bytes — 18 bytes / 6 chars dropped
  After:  10 chunks, 60 bytes — 0 bytes dropped

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Parent28e4761