Mosaic one model, many lenses

ADR 0014: Runtime knowledge sources — declared sources are the floor, runtime additions are the ceiling

  • Status: accepted
  • Date: 2026-10-01
  • Supersedes: nothing (extends the knowledge engine, K1; git sources, ADR 0013)

Context

Knowledge sources are today build-time facts: vector_dbs[].sources declares local paths and git repositories (ADR 0013), the CLI copies the knowledge/ directory into the output tree at build time, and the generated app ingests exactly those sources at boot. The runtime surface can only ingest inline text docs (POST /api/vectordb/{name}/ingest), delete one doc, and re-read the declared sources (reindex).

The use cases in sync/knowledge.md are operational, not build-time: an operator drops a book library (a folder of PDF/EPUB/DOCX files) or a git repository into the running app's data area and expects the app to index it — without re-rendering the app, redeploying, or editing the tessera. Conversely a source that should go away must be removable without its entries lingering in the index.

Constraints to respect:

  • ADR 0003 determinism: the declared model still renders the same bytes; runtime state lives in data files under the knowledge base, never in the rendered code.
  • ADR 0008/0010 lens split: the management surface is a web-lens feature (the /knowledge page) + REST; the static site lens gets a build-time projection of the declared sources only (a static page cannot show runtime state).
  • ADR 0012: the new routes join the route_policy table like every other route — default-deny authz applies (GET = view, mutations = update).
  • The source walk skips dot-dirs; anything the app writes under knowledge/ must live in a dot-dir so it can never be ingested by a source.

Decision

1. A persisted runtime registry under the knowledge base

<base>/knowledge/.runtime/sources.json (dot-dir ⇒ invisible to every source walk). Per collection: a map label → entry:

{
  "kb": {
    "legacyrepo2": {
      "source": { "path": "", "kind": "auto", "git_url": "knowledge/legacyrepo2.bundle", "git_ref": "main", "include": [], "exclude": [] },
      "slugs": ["legacyrepo2-inventory-c"],
      "files": 1,
      "chunks": 2,
      "added": "2026-10-01T07:00:00Z"
    }
  }
}
  • label is the stable per-source identity, computed by the engine (source_label): git sources use the ADR 0013 git_label (last URL segment, .git/.bundle stripped, slugified); path sources use the slugified last path segment. Labels are collision-checked against the declared sources of the collection — a runtime source can never shadow a declared one.
  • slugs are the entry slugs the add produced (the entry id minus the #chunk suffix). Removal deletes exactly those slugs — no prefix guessing, no damage to entries that happen to share a path prefix.
  • Boot re-ingests runtime sources (fail-soft, like declared sources), so additions survive restarts. reindex covers declared and runtime sources. Operators reset runtime state by deleting the .runtime dir.

2. REST surface (three routes, per collection)

  • GET /api/vectordb/{name}/sources — the source inventory: declared sources (recomputed live, as stats does today) plus runtime sources, each row carrying label, origin ("declared" | "runtime"), kind, files, chunks (+ git facts for git sources).
  • POST /api/vectordb/{name}/sources/add — body JSON { "path"?, "kind"?, "git": { "url", "ref"? } } or form-encoded fields path, kind, git_url, git_ref (the browser form). Fail-closed validation with named errors: unknown collection; neither path nor git url; kind outside auto|code|docs|pdf; a path source whose target does not exist; a git source while strategy.git is off; a label that collides with a declared or existing runtime source. On success: ingest (fail-closed — the caller wants an error, unlike boot's fail-soft), embed what is missing, persist the store, record the entry, return {label, files, chunks, slugs}.
  • DELETE /api/vectordb/{name}/sources/{label} — removes a runtime source (declared sources are rejected: "declared source — remove it from the tessera"), deletes its slugs, persists, answers JSON (agents / MCP).
  • POST /api/vectordb/{name}/sources/{label} — the no-JS remove form (same core, SSR-safe): answers 303 See Other to /knowledge?kb={name}&removed={label} (or …&source_error=…). A plain HTML form can only POST, so the form path is a second method on the same path — no client JS, works in full (SSR) with no hydration.

Content negotiation by body shape: the add handler answers JSON to a JSON body and, to a form-encoded body, 303 See Other to /knowledge?kb={name}&added={label} (or …&source_error=…) — so the same route serves agents (JSON) and the no-JS form (HTML navigation) without a second endpoint. All query values in a 303 Location are percent-encoded (a tiny percent_encode sits next to the percent_decode used for the form body — error messages carry spaces and quotes).

3. Surfaces

  • Web /knowledge page (all hydration modes): the per-KB sources table gains an origin column and, on runtime rows, a remove control — a plain POST form (no client JS, SSR-safe, so it works in full with no hydration and in csr/islands alike). Each KB card gains an add-source form (path or git url, ref, kind) posting to the negotiated add route. SSR mode renders a flash banner from the ?added= / ?removed= / ?source_error= query params.
  • Static site lens: a build-time knowledge sources page (/kb/, zola + html backends) projecting the declared collections and their sources (name, about, model, hybrid weights, strategy flags, source paths/git urls) — the ADR 0008 projection of the declared facts, with a nav entry. Runtime state is deliberately absent from the static lens.
  • MCP / CLI: unchanged — reindex_knowledge, knowledge_stats, repo_profile and mosaic knowledge-report now include runtime sources through the same extended sources/reports path (the registry is read by the generated app; the CLI report stays declared-only, as it runs without an app instance).

4. Engine additions (two fields, one function)

  • ingest::source_label(&Source) -> String (above).
  • IngestReport.label + SourceStats.label (serde-defaulted — existing persisted data and goldens stay valid).
  • SourceStats rows gain origin in the generated stats handler (it knows which labels are runtime), not in the engine — the engine stays source-shape-agnostic.

No new engine module, no new dependency, no new DSL key (runtime sources are data, not declaration).

Proof

  • Engine unit tests: source_label (git url, bundle, local path, nested path); IngestReport/SourceStats carry the label (serde default).
  • e2e (full-stack example, new committed fixture knowledge/legacyrepo2.bundle — a second tiny legacy repo with a distinctive token lumenledger):
    • GET …/sources lists the declared sources with origin: declared;
    • POST …/sources/add (git) adds legacyrepo2, and its content is hybrid-searchable;
    • POST …/sources/add (path) adds a directory source;
    • POST …/sources/legacyrepo2 (the no-JS remove form) answers 303 to /knowledge?removed=… and its entries go away;
    • DELETE …/sources/legacyrepo2 (the JSON path) removes it, 200;
    • fail-closed: adding a nonexistent path → 400; deleting a declared source → 400; a colliding label → 400;
    • the form-encoded add answers 303 to /knowledge?added=….
  • Conformance goldens re-pinned (new app handlers + routes, web page, site page, vendored engine); workspace tests + clippy -D warnings + fmt clean.

Consequences

  • A running app can grow and shrink its knowledge — book libraries and repositories included — from the page, the API, or an MCP-driven agent; the state persists across restarts and never leaks into the source walk or the rendered bytes.
  • Declared sources remain the source of truth: they cannot be removed at runtime, they cannot be shadowed by label, and the tessera still renders deterministically (ADR 0003).
  • The form/JSON negotiation adds one branch to the add handler; every other route stays JSON.
  • What is deliberately NOT built here: runtime edit of a runtime source's filters (re-add after delete is the documented cycle), remote (S3-style) source URIs (that is the data-access seam, a separate ADR), and multi-operator concurrency on the registry file (single process by design — the ADR 0011 posture).