Home

giogio / openez-graph

public
Overview Code History Branches Pull requestsIssuesInsights
main
HomeOverview Code PRsIssues

fix(indexer): scope token strategy and parser cache

3 months ago

378d208
Authored
asta-nguyen8/9/2026, 9:45:22 AM
Remove global mutable token-counting state (setFastTokenCount/_fastMode)
and introduce a scoped TokenCounter interface with fastTokenCounter
(chars/4, indexing) and exactTokenCounter (BPE, retrieval) instances.
Thread the counter through the indexer parse/chunk pipeline so indexing
uses the fast strategy without mutating global state, while retrieval
continues to use exact BPE counting.

Add resolveNativeParser() to resolve and cache the native tree-sitter
capability once. Graph builds now validate the parsed_documents cache
against the expected parser version (native-v1 when available,
fallback-v1 otherwise), so the second build reuses the fallback cache
with zero re-parses when native is unavailable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Parent4386de6

12 files changed
  • packages/core/src/tokenizer.ts+49−14
  • packages/indexer/src/index-workspace.ts+100−44
  • packages/indexer/src/languages.ts+53−28
  • packages/indexer/src/markdown.ts+18−6
  • packages/indexer/src/parsers/config-parser.ts+1−1
  • packages/indexer/src/parsers/fallback-parser.ts+4−1
  • packages/indexer/src/parsers/markdown-parser.ts+1−0
  • packages/indexer/src/parsers/tree-sitter-parser.ts
+15
−5
  • packages/indexer/src/parsers/types.ts+9−0
  • packages/indexer/src/tokenizer.ts+14−22
  • tests/parsed-documents-cache.test.ts+89−1
  • tests/tokenizer-concurrency.test.ts+55−0