Home

giogio / openez-graph

public
Overview Code History Branches Pull requestsIssuesInsights
main
HomeOverview Code PRsIssues

fix(p1): dynamic tokenizer import, source pairing, legacy marker, shared migration

3 months ago

d2ffd2b
Authored
asta-nguyen8/11/2026, 4:37:07 PM
P1 fixes from final review:

1. Make @huggingface/tokenizers a dynamic import in local-embedding.ts
   so embeddings remain optional. The static import forced the tokenizer
   dependency on all core package consumers. Type-only import for the
   Tokenizer type; runtime import inside ensureLoaded().

2. Give source arrays (callers, callees, relatedChunks, sources, files)
   minimum 1 in fitToTokenBudget so truncation drops whole result entries
   instead of emptying inner arrays. Context and structured sources stay
   paired under token truncation.

3. Check hasLegacyEmbeddings() before the embedding preflight in
   vectorSearch(). Legacy TEXT embeddings match the provider/model
   preflight but cannot be used with BLOB cosine search. The check
   prevents a wasted provider.embed API call before
   rankStoredEmbeddings discards the results.

4. Export migrateRegistrySchema from @openez-graph/db and use it in the
   web server. This eliminates the duplicated migrateRegistryColumns and
   adds the missing registry_meta table and one-shot graph invalidation
   backfill that the web server was not running.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Parented7e653

6 files changed
  • apps/mcp/src/mcp-core.ts+15−1
  • apps/web/src/server/sqlite.ts+6−53
  • packages/core/src/local-embedding.ts+3−2
  • packages/core/src/retrieval.ts+12−1
  • packages/db/src/sqlite/index.ts+2−0
  • packages/db/src/sqlite/registry-db.ts+107−99