Mosaic one model, many lenses

ADR 0018 — Vectordb RAG config + source-scoped search

Status: accepted

Context

The ee-pages portal and the EE platform improved their RAG stack (Sept–Oct 2026) along two axes that mosaic lacks:

  1. DSL-driven RAG config (ee-pages d59df40): the retrieval mode (hybrid/naive), the default top_k, and the chunking budget (chunk_size/chunk_overlap) are declared in the DSL per knowledge base, with env overrides and sensible defaults. Mosaic hard-codes chunk sizes (chunk.rs constants), has no retrieval-mode toggle (hybrid always), and takes top_k only per request (default 5 baked into every call site).
  2. Source-scoped search (ee-pages 5fa7bc9): every chunk carries a scope key (_source); search filters on it in both legs (vector + full-text); omitting the filter searches everything. Mosaic tracks source provenance only in meta["source"] (the raw path) — Store::hybrid, the REST search, the MCP search_knowledge tool, and the workflow search node have no way to restrict a query to one source.

Source scoping is the engine primitive behind the ee-pages single-collection portal (many apps, one collection, one filter) and is useful standalone: a collection with a books folder, a git repo, and a docs dir can answer "search only in the books" without post-hoc client-side filtering.

Decision

1. New vector_dbs[] DSL keys (per collection)

vector_dbs:
  - name: kb
    # retrieval: how queries are answered for this collection
    retrieval:
      mode: hybrid      # hybrid (default) | naive
      top_k: 5          # default top_k (a request may still override)
    # chunk: the markdown/docs chunking budget (chars)
    chunk:
      chars: 2000       # paragraph-group target (default, = today's constant)
      overlap: 0        # trailing chars re-emitted into the next group (default: none)

Semantics:

  • retrieval.mode:
    • hybrid (default) — today's behavior: BM25 + cosine fused by RRF.
    • naive — vector-only (cosine); degrades to keyword-only when no embedder is configured (fail-soft, exactly like today's no-key path).
  • retrieval.top_k — the collection's default top_k for every search surface (REST body, MCP tool, workflow node, assistant). A per-request value still overrides it.
  • chunk.chars / chunk.overlap — replace the hard constants PARAGRAPH_TARGET / (new) group overlap for the markdown/docs tier. Defaults (2000 / 0) are byte-identical to today's chunking, so no golden changes when the keys are omitted. The code-window, pdf-page, and book-chapter tiers are unchanged (their shape is structural, not a tunable budget).
  • Plan validation (fail-closed): mode ∈ {hybrid, naive}; top_k ≥ 1; chars ≥ 1; overlap < chars.

Env overrides: none for now (the DSL is the declared floor; ADR 0017's runtime overlay pattern applies to models, and retrieval tuning is per-collection declaration — keeping the surface small).

  • Engine: Entry.meta gains a stable source_label stamped at ingest — the same source_label(src) already used for runtime-source registry identity (git → URL slug, data URI → URI label, path → slugified segment). Inline DSL docs get source_label: "dsl" (their meta.source already is). The raw path stays in meta.source (display/provenance); the label is the filter key (stable across path moves is out of scope — labels are path-derived today by design).
  • Store::hybrid(...) gains a source: Option<&str> parameter: when set, both legs are restricted to entries whose meta.source_label equals it (pre-filtered candidate pools, not post-filtering — a rare source must not be crowded out of the top-N candidate window). None = today's behavior.
  • Surfaces (all gain an optional source):
    • POST /api/vectordb/{name}/search — body {query, top_k?, source?}.
    • MCP search_knowledge — optional source argument.
    • Workflow search node — optional source attribute.
    • The assistant chat endpoint is intentionally global (the portal assistant answers across the corpus); scoping it is a later knob.

3. Generated code shape (OOP posture)

The generated app keeps one knowledge_retrieval(name) function as the single source of per-collection tuning (extending its return tuple to (vw, bw, mmr, rerank, mode, top_k)), and vector_search(name, query, top_k, source) as the single search entry used by REST, MCP, workflow, and assistant alike — one function, many bridges, as in ADR 0014/0016.

Consequences

  • New keys are additive; omitted → byte-identical behavior and goldens.
  • Store::hybrid's signature changes (crate-internal + generated call sites); the goldens re-pin (the generated ws.rs/server.rs text changes).
  • meta.source_label lands in persisted indexes; old indexes without it simply can't be filtered on (a label-filtered search over a pre-0018 index returns nothing for that label until reindex — acceptable; reindex is cheap and idempotent).
  • ee-pages' single shared collection layout is not ported as such: mosaic's per-collection stores + the new source filter cover the same multi-scope need more idiomatically (a collection is a scope, and sources within it are sub-scopes).
  • Follow-ups (separate ADRs): per-source delta re-indexing (manifest, skip-unchanged), the persisted document knowledge graph with graph-aware retrieval (EE's ee-knowledge work), assistant-level source scoping.