feat(bridge): incremental index with content-hash cache + bounded parallel embed
Why: The DeepInfra (BAAI/bge-large-en-v1.5) switch raised per-batch embedding latency ~2x vs OpenAI text-embedding-3-small. The bridge's serial BATCH=10 loop on a 251-chunk vault was running ~65-80s and getting killed by Netlify's 60s sync- function cap, leaving the UI with a 504 + HTML body. PR #202 added timing logs that confirmed embed_batch was the dominant step.
What: 1. Content-hash cache. Each chunk gets a versioned (v1:) SHA-256 of its text + canonical metadata (sorted tags/entity, null-equivalent missing fields). The sqlite-vec table gains +content_hash and +chunk_id auxiliary columns; on re-index the bridge calls store.getChunkHashes(vaultId), partitions chunks into {toEmbed, skippedCached, orphans}, and only embeds + upserts the diff. Orphans (chunk_ids in the store but absent from the current canister export) are removed via store.deleteByChunkIds — preserves the search-orphan invariant the phase3 contract test was protecting. 2. Bounded-concurrency embed pool. lib/parallel-embed-pool.mjs gives the bridge runWithConcurrency(thunks, {concurrency, onSettled}) with order-preserving results and fail-fast semantics. INDEXER_EMBED_BATCH_SIZE (default 50, was 10) and INDEXER_EMBED_CONCURRENCY (default 5) are env-tunable; this drops a 251- chunk full re-embed from ~65-80s to ~10-15s under the same DeepInfra latency. 3. DeepInfra 429 backoff. embedDeepInfraWithUsage now retries once on 429, honoring Retry-After (capped to 5s so retries can't strand the function cap). fetchImpl + sleepFn injection keeps tests hermetic. 4. Frontend stops doubling the bridge call. web/hub/hub.js api() learns noRetry: true (zero retries when set; flag is stripped before forwarding to fetch). The Re-index button passes noRetry: true so a 30s gateway timeout no longer fires a second bridge index in parallel. Toast surfaces (N embedded, M cached) when the cache hit rate is non-zero.
Schema migration: sqlite-vec virtual tables can't be ALTERed, so ensureCollection drops + recreates when the existing table lacks vault_id (pre-existing logic) OR content_hash (new). Net cost: a one-time full re-embed on the first re-index after merge; every subsequent re-index hits the cache.
Mirror surface on Qdrant store (lib/vector-store.mjs): getChunkHashes via paginated scroll; deleteByChunkIds via point-id delete. Both backends are interchangeable from the bridge's perspective.
Tests: 1729 -> 1797 (+68). 0 fail. Pure helpers + injectable fetch keep all new tests deterministic and hermetic.
Refs: PR #202 (per-step timing logs), netlify.toml gateway/bridge timeout 60s.
0 comments
muse hub commit comment sha256:14eb3b29881673f444c792522a4a90f47df786844360a05a3c44b86f2ab69ed8 --body "your comment"
No comments yet. Be the first to start the discussion.