Mosaic one model, many lenses

ADR 0019 — Delta re-indexing: per-source file manifests

Status: accepted

Context

Every reindex (and every boot re-walk) in mosaic re-ingests all files of all sources and resets every entry's embedding to None, so the embedder re-runs on the whole corpus — even when nothing changed. The ee-pages portal fixed exactly this (its doc_files manifest + delta ingest on publish): each source keeps a manifest of relpath → content-hash; on re-index only the added and changed files are chunked + embedded, removed files are deleted from the index, and unchanged files keep their entries and their embeddings. The manifest self-heals: a corrupted or missing manifest simply re-ingests the whole source once.

For a book library or a large git source this is the difference between a reindex that costs unchanged files ≈ 0 and one that costs the whole corpus of embedding calls.

Decision

1. Per-source manifests in the engine

mosaic-knowledge gains:

  • file_hash(data) -> String — sha256 hex of the file bytes (the hash is of the content, so a touched-but-identical file is unchanged).
  • SourceManifest { files: BTreeMap<String, String> } — relpath → hash, per source label, serde-shaped (it rides the same persistence seam as the store).

2. The delta path

A new engine entry point, ingest_source_delta(src, base, strategy, env, previous: Option<&SourceManifest>, store: &mut Store) -> (IngestReport, SourceManifest):

  1. Walk the source (same walk + filters as today).
  2. For each file: hash the bytes.
    • hash present in previous and the doc slug already in the store → skip (entry + embedding untouched).
    • new or changed → process_file as today, store.upsert(slug, …) (fresh entries get emb: None; the caller's embed_missing pass embeds exactly the new ones).
    • in previous but absent now → store.remove_slug(slug) (the file left the source).
  3. Return the new manifest (the caller persists it).

A missing/empty previous (first run, manifest lost, label renamed) is exactly today's full ingest — self-healing, no migration.

3. Manifest persistence

One file per collection, next to the runtime registry, through the same data seam (local knowledge/.runtime/manifest.json; a data-URI persist lands it under <persist-dir>/.runtime/manifest.json):

{ "<collection>": { "<source-label>": { "<relpath>": "<sha256>" } } }
  • Boot ingest (declared + runtime sources) and POST /reindex use the delta path and rewrite their labels' manifests.
  • POST /sources/add records the new label's manifest.
  • DELETE /sources/{label} drops the label's manifest (the slugs are already removed via the registry).

4. What does NOT change

  • Inline DSL docs (VECTORDB_SEEDS): idempotent by id, tiny — no manifest.
  • Deterministic ids/slugs (unchanged — that is what makes the per-file skip and remove work).
  • The store shape, the REST surface, the e2e contract.
  • The git checkout cache (knowledge/.gitcache/<label>): a reindex still reuses the work tree (a git source's files are whatever the checkout holds; the manifest decides what of that is new).

Consequences

  • Reindex cost drops from O(corpus) embedding calls to O(changed); boot with a warm index + unchanged tree re-embeds nothing.
  • store.remove_slug must stay exact (a label's manifest only ever removes that label's slugs — no cross-source damage).
  • A content change that does not change chunk boundaries still re-upserts (new emb: None → re-embedded) — correct, slightly wasteful vs a chunk-level diff; out of scope.
  • Old indexes/manifests from pre-0019 builds simply have no manifest file → one full ingest on first boot, then delta forever.