perf(index): optimize FTS, parse pipeline, and graph accuracy 3 months ago
Authored
JoeJoeflyn 8/6/2026, 5:33:02 PM - Replace ts-morph with oxc-parser for TS/JS (5x faster parse)
- Skip AST walk during indexing (parse only for chunking, not graph)
- Move FTS insert after main transaction with triggers down
- Truncate FTS search_text to 400 chars
- page_size=16384 at DB creation (fewer page splits)
- journal_mode=MEMORY + locking_mode=EXCLUSIVE during bulk write
- mmap_size=512MB for larger memory-mapped I/O
- Move setOptimizedWriteMode(false) after finalize to avoid
mid-finalize fsync flushing 70MB of dirty pages
- Use PASSIVE checkpoint to avoid blocking fsync on small repos
- Migrate TEXT PKs to INTEGER PRIMARY KEY (auto-increment)
- Use RETURNING id for graph node batch insert (fixes empty graph bug)
- Add C to graph builder language filter
- Add TS/JS graph building via OxcParser with call/import extraction
- Fix function pointer typedef name extraction (compare_fn_t not a)
- Skip struct/union/enum type references without body (no false symbols)
- Keep qualified call names (obj->method instead of method)
- Remove symbol/call caps in graph builder (was 30 symbols, 20 calls)
- Fix OxcParser symbol extraction for export wrappers and variable decls
- Add import edge resolution (file-to-file graph edges)
- Switch hashContent from SHA-256 to Bun.hash (xxHash64, 2x faster)
- Linux kernel: 88s -> 23.9s (3.7x faster) Parentsdfed446 18a688d
115 files changed
.agents/skills/superpowers-brainstorming/SKILL.md +152 −0 .agents/skills/superpowers-brainstorming/scripts/frame-template.html +213 −0 .agents/skills/superpowers-brainstorming/scripts/helper.js +179 −0 .agents/skills/superpowers-brainstorming/scripts/server.cjs +723 −0 .agents/skills/superpowers-brainstorming/scripts/start-server.sh +209 −0 .agents/skills/superpowers-brainstorming/scripts/stop-server.sh +120 −0 .agents/skills/superpowers-brainstorming/spec-document-reviewer-prompt.md +49
−0
.agents/skills/superpowers-brainstorming/visual-companion.md +312 −0
.agents/skills/superpowers-dispatching-parallel-agents/SKILL.md +177 −0
.agents/skills/superpowers-executing-plans/SKILL.md +70 −0
.agents/skills/superpowers-finishing-a-development-branch/SKILL.md +201 −0
.agents/skills/superpowers-receiving-code-review/SKILL.md +218 −0
.agents/skills/superpowers-requesting-code-review/SKILL.md +102 −0
.agents/skills/superpowers-requesting-code-review/code-reviewer.md +173 −0
.agents/skills/superpowers-subagent-driven-development/SKILL.md +506 −0
.agents/skills/superpowers-subagent-driven-development/implementer-prompt.md +142 −0
.agents/skills/superpowers-subagent-driven-development/re-review-prompt.md +107 −0
.agents/skills/superpowers-subagent-driven-development/scripts/review-package +46 −0
.agents/skills/superpowers-subagent-driven-development/scripts/sdd-workspace +40 −0
.agents/skills/superpowers-subagent-driven-development/scripts/task-brief +41 −0
.agents/skills/superpowers-subagent-driven-development/task-reviewer-prompt.md +186 −0
.agents/skills/superpowers-systematic-debugging/CREATION-LOG.md +133 −0
.agents/skills/superpowers-systematic-debugging/SKILL.md +290 −0
.agents/skills/superpowers-systematic-debugging/condition-based-waiting-example.ts +158 −0
.agents/skills/superpowers-systematic-debugging/condition-based-waiting.md +120 −0
.agents/skills/superpowers-systematic-debugging/defense-in-depth.md +128 −0
.agents/skills/superpowers-systematic-debugging/find-polluter.sh +72 −0
.agents/skills/superpowers-systematic-debugging/root-cause-tracing.md +183 −0
.agents/skills/superpowers-systematic-debugging/test-academic.md +14 −0
.agents/skills/superpowers-systematic-debugging/test-pressure-1.md +65 −0
.agents/skills/superpowers-systematic-debugging/test-pressure-2.md +75 −0
.agents/skills/superpowers-systematic-debugging/test-pressure-3.md +75 −0
.agents/skills/superpowers-test-driven-development/SKILL.md +334 −0
.agents/skills/superpowers-test-driven-development/writing-good-tests.md +198 −0
.agents/skills/superpowers-using-git-worktrees/SKILL.md +170 −0
.agents/skills/superpowers-using-superpowers/SKILL.md +62 −0
.agents/skills/superpowers-using-superpowers/references/antigravity-tools.md +23 −0
.agents/skills/superpowers-using-superpowers/references/codex-tools.md +39 −0
.agents/skills/superpowers-using-superpowers/references/gemini-tools.md +63 −0
.agents/skills/superpowers-using-superpowers/references/pi-tools.md +16 −0
.agents/skills/superpowers-verification-before-completion/SKILL.md +127 −0
.agents/skills/superpowers-writing-plans/SKILL.md +175 −0
.agents/skills/superpowers-writing-plans/plan-document-reviewer-prompt.md +49 −0
.agents/skills/superpowers-writing-skills/SKILL.md +741 −0
.agents/skills/superpowers-writing-skills/anthropic-best-practices.md +1173 −0
.agents/skills/superpowers-writing-skills/examples/CLAUDE_MD_TESTING.md +200 −0
.agents/skills/superpowers-writing-skills/graphviz-conventions.dot +172 −0
.agents/skills/superpowers-writing-skills/persuasion-principles.md +220 −0
.agents/skills/superpowers-writing-skills/render-graphs.js +171 −0
.agents/skills/superpowers-writing-skills/testing-skills-with-subagents.md +404 −0
.devin/hooks.v1.json +14 −0
.devin/hooks/superpowers-session-start.sh +46 −0
.gitignore +3 −0
AGENTS.md +5 −4
CHANGELOG.md +65 −0
README.md +1 −1
apps/cli/package.json +5 −3
apps/cli/src/cli.ts +206 −13
apps/cli/tsup.config.ts +36 −5
apps/mcp/src/mcp-core.ts +94 −6
apps/web/src/lib/api.ts +18 −1
apps/web/src/routes/workspaces/index.tsx +276 −46
apps/web/src/server/index.ts +36 −6
apps/web/src/server/sqlite.ts +94 −5
docs/superpowers/plans/2026-08-06-workspace-remove-pin.md +1220 −0
docs/superpowers/specs/2026-08-06-workspace-remove-pin-design.md +177 −0
landing/src/app/page.tsx +17 −4
packages/core/src/graph.ts +69 −36
packages/core/src/tokenizer.ts +8 −1
packages/db/src/sqlite-index.ts +2 −0
packages/db/src/sqlite/database-loader.ts +1 −20
packages/db/src/sqlite/index.ts +2 −0
packages/db/src/sqlite/registry-db.ts +53 −3
packages/db/src/sqlite/remove-workspace.ts +118 −0
packages/db/src/sqlite/repository.ts +123 −92
packages/db/src/sqlite/schema.ts +2 −0
packages/db/src/sqlite/types.ts +122 −18
packages/db/src/sqlite/workspace-db.ts +64 −28
packages/indexer/package.json +7 −1
packages/indexer/src/hash.ts +5 −6
packages/indexer/src/index-workspace.ts +349 −270
packages/indexer/src/languages.ts +589 −529
packages/indexer/src/markdown.ts +4 −3
packages/indexer/src/parse-core.ts +0 −76
packages/indexer/src/parse-worker.ts +0 −70
packages/indexer/src/parsers/config-parser.ts +26 −0
packages/indexer/src/parsers/fallback-parser.ts +42 −0
packages/indexer/src/parsers/index.ts +7 −0
packages/indexer/src/parsers/markdown-parser.ts +30 −0
packages/indexer/src/parsers/oxc-parser.ts +243 −0
packages/indexer/src/parsers/parser-registry.ts +51 −0
packages/indexer/src/parsers/tree-sitter-parser.ts +86 −0
packages/indexer/src/parsers/types.ts +64 −0
packages/indexer/src/scanner.ts +74 −88
packages/indexer/src/tree-sitter/configs.ts +548 −0
packages/indexer/src/tree-sitter/index.ts +4 −0
packages/indexer/src/tree-sitter/loader.ts +134 −0
packages/indexer/src/tree-sitter/parse.ts +282 −0
packages/native/Cargo.lock +522 −0
packages/native/Cargo.toml +22 −0
packages/native/index.d.ts +43 −0
packages/native/index.js +295 −0
packages/native/index.linux-x64-gnu.node — —
packages/native/index.node — —
packages/native/package-lock.json +32 −0
packages/native/package.json +13 −0
packages/native/src/lib.rs +260 −0
packages/native/src/parser.rs +992 −0
pnpm-lock.yaml +377 −71
tests/mcp-tools.test.ts +52 −0
tests/parser-registry.test.ts +157 −0
tests/registry.test.ts +26 −0
tests/remove-workspace.test.ts +77 −0
tests/tree-sitter-parser.test.ts +379 −0
tests/web-workspaces-endpoint.test.ts +108 −0