Home

giogio / openez-graph

public
Overview Code History Branches Pull requestsIssuesInsights
main
HomeOverview Code PRsIssues

perf(indexer): streaming DB writes, chunk merging, adaptive workers

3 months ago

b49e165
Authored
JoeJoeflyn8/4/2026, 10:01:01 AM
- Switch from bun:sqlite back to better-sqlite3 (FTS5 5x faster)
- Streaming prepared statements instead of dynamic bulk SQL
- Merge small chunks (<30% of target) to reduce chunk count 4x
- Adaptive worker count based on file count (min of cores, files/50)
- Skip graph building on cold index (built lazily on first query)
- Skip SHA256 on cold index (use mtime:size as fingerprint)
- FTS backfill via single INSERT...SELECT instead of per-row inserts
- Buffer-based stripNonCode state machine (35% faster than regex)
- Fast tokenization approximation (chars/4) replacing gpt-tokenizer
- Cached batch timestamp for streaming inserts

Zed (2903 files): 33.6s -> 8.5s cold, 2.8s -> 1.0s warm

Parentdacdda1

19 files changed
  • apps/cli/package.json+5−5
  • apps/cli/tsup.config.ts+3−3
  • apps/web/package.json+2−2
  • brain.config.ts+1−1
  • packages/config/src/types.ts+2−2
  • packages/core/src/graph.ts+2−2
  • packages/core/src/retrieval.ts+2−2
  • packages/db/package.json+2−4
  • packages/db/src/sqlite/repository.ts+57−2
  • packages/db/src/sqlite/types.ts+21−0
  • packages/indexer/src/code.ts+44−12
  • packages/indexer/src/index-workspace.ts+403−275
  • packages/indexer/src/languages.ts+102−22
  • packages/indexer/src/markdown.ts+1−1
  • packages/indexer/src/parse-core.ts+1−0
  • packages/indexer/src/parse-worker.ts+36−2
  • packages/indexer/src/scanner.ts+84−24
  • packages/indexer/src/tokenizer.ts+22−0
  • pnpm-lock.yaml+33−235