feat: FTS5 + graph retrieval, remove embeddings, perf and token optimization
4 months ago
Authored
JoeJoeflyn- Replace embeddings with FTS5 full-text search
- Add graph retrieval with RRF fusion
- Replace ts-morph with native TypeScript compiler APIs + regex parsing
- Use length/4 token approximation instead of gpt-tokenizer
- Wrap indexing loop in a single SQLite transaction
- Lazy getText() and skip import traversal for files without exports
- Batch FTS inserts + sync DB methods for hot path
- Add throttled catch-up indexing to MCP read tools
- Make file watcher opt-in
- Bundle frontend with CLI, change default web port to 11368
- Add CLI publishing guide and npm distribution setup
- Add MIT license and improve CLI package metadata
- Add Python indexing support with call graph extraction
- Add workspace root path validation, bump CLI to 0.3.2
- Create graph_runs records during indexing, update registry graphStatus
- Add memories_fts table, memory_query searches memories alongside chunks
- Remove unused files, deps, and duplicate UI components
Index time: 1.58s → 0.88s wall clock for 163 files. Bundle: 14.86 MB → 12.19 MB.
Parentd1d2f38
103 files changed
- .claude/CLAUDE.md+379−0
- .env.example+6−8
- .gitignore+3−0
- CHANGELOG.md+31−0
- LICENSE+21−0
- README.md+112−36
- apps/cli/.gitignore+2−0
- apps/cli/LICENSE+21−0
- apps/cli/README.md
+112
−0
apps/cli/package.json+54−4apps/cli/src/cli.ts+63−11apps/cli/src/mcp-bridge.ts+1−1apps/cli/src/resolve-cli.ts+54−0apps/cli/src/setup-claude.ts+5−5apps/cli/src/setup-codex.ts+4−5apps/cli/src/setup-devin.ts+71−0apps/cli/src/setup-opencode.ts+5−5apps/cli/src/web-server.ts+4−0apps/cli/tsdown.config.ts+55−0apps/cli/tsup.config.ts+51−0apps/mcp/package.json+3−2apps/mcp/src/mcp-core.ts+120−0apps/web/package.json+9−3apps/web/src/components/chunk-boundary-bar.tsx+83−0apps/web/src/components/chunk-content.tsx+119−0apps/web/src/components/documents/documents-columns.tsx+72−0apps/web/src/components/documents/documents-data-table.tsx+159−0apps/web/src/components/documents/documents-filters.tsx+81−0apps/web/src/components/graph/GraphClient.tsx+88−7apps/web/src/components/graph/WorkspaceGraph.tsx+203−123apps/web/src/components/indexing-progress.tsx+146−0apps/web/src/components/memory-detail-panel.tsx+118−0apps/web/src/components/sidebar.tsx+25−14apps/web/src/components/structured-content-viewer.tsx+133−0apps/web/src/components/workspace-selector.tsx+71−0apps/web/src/globals.css+10−10apps/web/src/lib/api.ts+155−6apps/web/src/lib/constants.ts+114−0apps/web/src/lib/pagination.tsx+11−4apps/web/src/lib/queries.ts+128−20apps/web/src/lib/use-sse.ts+189−0apps/web/src/lib/utils.ts+8−0apps/web/src/main.tsx+5−2apps/web/src/routeTree.gen.ts+66−0apps/web/src/routes/__root.tsx+56−2apps/web/src/routes/documents.tsx+105−35apps/web/src/routes/index.tsx+27−18apps/web/src/routes/memories.tsx+140−0apps/web/src/routes/query.tsx+408−97apps/web/src/routes/settings.tsx+50−39apps/web/src/routes/workspaces/$workspaceId/documents/$documentId/chunks.tsx+307−0apps/web/src/routes/workspaces/$workspaceId/graph.tsx+4−4apps/web/src/routes/workspaces/$workspaceId/index.tsx+413−68apps/web/src/routes/workspaces/$workspaceId/symbols.tsx+239−0apps/web/src/routes/workspaces/index.tsx+4−2apps/web/src/routes/workspaces/new.tsx+5−4apps/web/src/server/index.ts+573−47apps/web/src/server/run-tracker.ts+112−0apps/web/src/server/sqlite.ts+753−24apps/web/src/server/start.ts+10−0apps/web/vite.config.ts+3−2apps/worker/src/cli.ts+0−3install.ps1+74−0install.sh+94−0package.json+2−6packages/config/src/env.ts+7−7packages/config/src/load-brain-config.ts+4−14packages/core/package.json+1−5packages/core/src/index.ts+0−1packages/core/src/retrieval.ts+45−44packages/core/src/tokenizer.ts+13−5packages/core/src/types.ts+12−0packages/db/package.json+1−7packages/db/src/index.ts+0−3packages/db/src/sqlite/database-loader.ts+5−1packages/db/src/sqlite/index.ts+1−1packages/db/src/sqlite/repository.ts+380−77packages/db/src/sqlite/schema.ts+2−15packages/db/src/sqlite/types.ts+132−20packages/db/src/sqlite/workspace-db.ts+74−11packages/indexer/package.json+1−9packages/indexer/src/code.ts+218−106packages/indexer/src/index-workspace.ts+271−159packages/indexer/src/languages.ts+68−15packages/indexer/src/markdown.ts+1−1packages/indexer/src/types.ts+0−9packages/queue/src/index.ts+0−1packages/ui/package.json+4−1packages/ui/src/components/badge.tsx+1−1packages/ui/src/components/button.tsx+2−1packages/ui/src/components/card.tsx+1−1packages/ui/src/components/code-block.tsx+1−1packages/ui/src/components/input.tsx+1−1packages/ui/src/components/label.tsx+1−1packages/ui/src/components/select.tsx+175−13packages/ui/src/components/table.tsx+1−1packages/ui/src/components/textarea.tsx+1−1packages/ui/src/components/ui/alert-dialog.tsx+84−0packages/ui/src/components/ui/sheet.tsx+1−1packages/ui/src/components/ui/sidebar.tsx+2−2packages/ui/src/components/ui/slider.tsx+69−0packages/ui/src/index.ts+3−1tsconfig.base.json+3−1