3 months ago
378d208Remove global mutable token-counting state (setFastTokenCount/_fastMode) and introduce a scoped TokenCounter interface with fastTokenCounter (chars/4, indexing) and exactTokenCounter (BPE, retrieval) instances. Thread the counter through the indexer parse/chunk pipeline so indexing uses the fast strategy without mutating global state, while retrieval continues to use exact BPE counting. Add resolveNativeParser() to resolve and cache the native tree-sitter capability once. Graph builds now validate the parsed_documents cache against the expected parser version (native-v1 when available, fallback-v1 otherwise), so the second build reuses the fallback cache with zero re-parses when native is unavailable. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Parent4386de6