Mosaic one model, many lenses

ADR 0015: OpenDAL data seam — knowledge reads from, and persists to, any data backend

  • Status: accepted
  • Date: 2026-10-01
  • Supersedes: nothing (extends the vector_dbs[].sources[].path and vector_dbs[].persist constructs, K1; the runtime-source registry, ADR 0014)

Context

The knowledge engine reads sources exclusively from the local filesystem (std::fs walk + read) and persists state exclusively to local JSON files (the index via Store::save/load, the ADR 0014 runtime registry next to it). The requirement: support OpenDAL in mosaic, to be able to read and write data from different data sources/targets — concretely for the knowledge feature: a book library that lives on object storage must be addable as a source (declared or at runtime, ADR 0014), and the knowledge state (index + runtime registry) must be writable to a target beyond the local disk (object storage that survives container replacement, in-RAM for tests).

Apache OpenDAL is the accepted data-access layer for exactly this: one pure-Rust API over many backends (fs, memory, s3, gcs, azblob, oss, obs, cos, hdfs, …), services feature-gated.

Three constraints shape the design:

  1. The engine is sync, and it is called from mixed contexts. ingest_source, persist I/O and the registry I/O run at boot (inside the host app's async runtime) and in tokio::task::spawn_blocking workers (no runtime context at all). OpenDAL's sync BlockingOperator must be constructed inside a runtime context (it captures the current Handle) — which the spawn_blocking contexts do not have. The seam therefore drives OpenDAL's async Operator from the engine's own dedicated tokio runtime (data::runtime(), a small multi-thread runtime created lazily once) via block_on — valid from every calling context (a separate runtime, so no nested-block_on panic; blocking the calling thread matches the engine's existing blocking-I/O posture).
  2. The engine stays env-pure (Z7). The workspace bans std::env::var in the engine/generator crates — so storage credentials are not read by the engine. The generated app (and the mosaic knowledge-report CLI) wire a StorageEnv provider over their own env reads; the engine resolves credentials through the provider and fails closed with a named error when it has none.
  3. The vendored engine is compiled into the web lens in full/islands (native SSR) only — never for wasm (csr). A native-only dependency is therefore safe to add to the engine.

Decision

One engine module (a sync seam), two URI upgrades, env-gated credentials, no new DSL key beyond reusing path/persist as URIs:

  1. A new engine module data — the sync DataBackend trait.

    pub trait DataBackend: Send + Sync {
    /// Direct children of a directory (`""` = the backend root).
    fn list(&self, dir: &str) -> io::Result<Vec<DataEntry>>;
    fn read(&self, path: &str) -> io::Result<Vec<u8>>;
    /// Write bytes (parents as the service supports).
    fn write(&self, path: &str, data: &[u8]) -> io::Result<()>;
    fn exists(&self, path: &str) -> bool;
    fn is_dir(&self, path: &str) -> bool;
    /// Human-readable location (stats / the /kb page).
    fn describe(&self) -> String;
}
pub struct DataEntry { pub name: String, pub is_dir: bool }

/// Storage credentials / region / endpoint provider (see §4).
pub struct StorageEnv { /* Arc<dyn Fn(mosaic_key, standard_key) -> Option<String>> */ }

    Two implementations + a factory:

    • LocalBackend { root: PathBuf } — std::fs. The default: a plain path resolves to this, byte-identical to today's behavior (all existing engine tests stay green unchanged).
    • OpenDalBackend { op: Operator, desc } — any compiled-in OpenDAL service, driven by the engine's dedicated runtime (constraint 1); list = per-directory op.list (OpenDAL lists the queried directory itself + its direct children — the walker skips the self-entry), recursion stays in the ingest walker.
    • data::backend_for(uri: &str, base: &Path, env: &StorageEnv) -> Result<(Box<dyn DataBackend>, String), String> — the backend is rooted at the URI's parent, the returned rel addresses the URI's last segment ("" = the whole root): sources walk from rel, the persist target reads/writes rel directly.
      • no scheme (or file://) → LocalBackend rooted at base (relative) or the file's parent (absolute);
      • s3://bucket/…, memory://…, fs://host/… → OpenDalBackend; any other scheme is a named error (data: scheme x is not available in this build); a service that cannot be configured (missing credentials/region) is a named error, fail-closed (the caller wanted a verdict).

  2. Sources can be data URIs (read from any target). sources[].path accepts an OpenDAL URI (s3://bucket/books/, memory://lib, fs://host/lib) alongside a plain path and a git source (unchanged — git stays on the git-CLI seam). Ingestion resolves the source's backend via backend_for and walks with list/read instead of the fs walk. Locators/meta for a URI source carry <label>/<rel-from-source-root> — the K8 git-label pattern: label is the slugified scheme-authority-path of the URI (deterministic, independent of any cache location); entry ids are the flat slug of that path (the existing id convention). A single-file URI (s3://bucket/book.epub) ingests as one document; include/exclude filters are unchanged (substring match on the rel path). Remote PDFs are staged to a temp file for pdf-extract (it takes an fs path) and cleaned up after. ADR 0014's source_label for a URI source is that label — the runtime add/remove flow (K9) works identically for remote data: the /knowledge form's path input accepts URIs, and the fail-closed "path not found" check becomes !backend.exists(root) (the engine performs it; the app's fs pre-check skips URIs).

  3. The persist target can be a data URI (write to any target). vector_dbs[].persist (existing key, today a local path) accepts an OpenDAL URI (e.g. s3://bucket/kb.json). The engine gains byte-level Store::serialize() -> Vec<u8> / Store::deserialize(&[u8]) -> Store (the existing Persisted shape); save(Path) / load(Path) remain as LocalBackend shims so engine tests and the CLI are unchanged. The generated app resolves the persist backend from the URI and reads/writes the index and the ADR 0014 runtime registry through it; the registry sits next to the index — local: knowledge/.runtime/sources.json (unchanged), URI target: <target-dir>/.runtime/sources.json.

  4. Env-gated credentials through a StorageEnv provider — the DSL declares placement, env declares the provider (the established pattern; secrets never in tessera.yaml): MOAIC_STORAGE_{SCHEME}_{KEY} (e.g. MOAIC_STORAGE_S3_ACCESS_KEY_ID, …_SECRET_ACCESS_KEY, …_REGION, …_ENDPOINT; scheme upper-cased, -→_), falling back to the provider-standard env (AWS_* for s3). Because the engine is env-pure (constraint 2), the env reads live in the generated app (knowledge_storage_env(), a lazily-built StorageEnv::new closure over std::env::var) and in the knowledge-report CLI; the engine only calls the provider (env.lookup(mosaic_key, standard_key)). s3 requires key + secret + region (fail-closed named errors when absent — the builder validates); no credentials are read for local paths, memory://, or fs://.

  5. Dependency. opendal 0.59 (default-features = false, features services-fs, services-memory, services-s3) plus tokio (rt-multi-thread, net, time — the engine's dedicated runtime, constraint 1), in the workspace crate and the vendored knowledge-engine/Cargo.toml. Pure Rust (rustls http transport; no native linking, no downloads at build time — same posture as the K7 ort load-dynamic seam). Additional backends (gcs, azblob, oss, …) are an enable-a-feature + one build_operator match arm change.

  6. Fail-soft / fail-closed split (the established posture).

    • Boot: an unreachable URI source or persist target → a named error for that source/target; the app boots; the remaining sources/index work (fail-soft, exactly like K8 git sources).
    • K9 runtime add of a URI source → fail-closed 400 with the named error (the caller wants a verdict).
    • memory:// is per-instance RAM: each backend_for call builds a fresh, private in-memory store — it is a zero-config test/dev seam (deterministic: always empty on a fresh process), not a shared or durable target.

Proof

  • Engine unit tests (new data module tests + ingest tests, hermetic — no network):
    • memory:// round-trip through the seam: write / read / list / exists;
    • ingest from a data URI (fs:// over a temp dir — the real OpenDAL path, hermetic): directory source (files ingested, locators <label>/<rel>, content asserted) and single-file source;
    • missing URI roots fail closed (memory:// — always fresh/empty — and a nonexistent fs:// dir both → knowledge source not found);
    • Store::serialize/deserialize round-trip (entries + citations; corrupt bytes → empty store);
    • backend_for resolution: plain path → local (unchanged), unknown scheme → named error, s3 without credentials → named error, s3 without region → named error (all before any I/O), s3 with an explicit StorageEnv provider resolves;
    • every pre-existing engine test green unchanged (the local default).
  • e2e (full-stack): a declared memory://kb-remote source on kb — boot is fail-soft (the app is healthy, the other declared sources remain hybrid-searchable), the URI source is listed in /api/vectordb/kb/sources (proving declared-URI plumbing end to end), and a runtime POST …/sources/add of memory://does-not-exist is fail-closed 400 "not found" (proving the K9 URI path deterministically — memory is always fresh/empty, so no cloud is involved). Posture per K7/K8: the seam's happy path is unit-tested hermetically; CI asserts the fail-soft/fail-closed contracts; no live cloud in CI.
  • Conformance goldens re-pinned (vendored engine + generated persist plumbing); workspace tests + clippy -D warnings + fmt clean.

Consequences

  • A mosaic app can read knowledge from and write knowledge state to any OpenDAL backend — a book library on S3 is a one-line source (declared or added at runtime, ADR 0014), and the index + runtime registry can live on a target that outlives the container.
  • Zero behavior change by default: every existing plain-path source and persist resolves to LocalBackend (byte-identical I/O); no new DSL key (path/persist are simply URIs now); no new env for local apps.
  • The engine gains one small, sync, well-tested seam (a trait + two impls + a factory) — the OOP shape the platform's external seams are moving toward (git CLI, OCR, rerank, ONNX).
  • New dependency: opendal (pure Rust, feature-gated services; native-only lenses compile it — the wasm/CSR web lens never does).