ADR 0024 — Embedded storage tier: sqlite + lancedb, scale-based engine defaults

Ports the EE storage-tier model into the tessera authoring format, the deploy lens, and the knowledge engine: small apps run fully embedded (sqlite + lancedb), no external store workloads at all.

Context

EE's model. In EE (blocks/ai/ee-vectordb, doc/reference/dsl/deploy.md), LanceDB is an embedded, in-process vector database (like SQLite/DuckDB): no pod, no Service, no image. The deploy DSL's lancedb { } block mounts a PersistentVolumeClaim onto the app's own pod and sets VECTOR_DB_URL to the local mount path (never a network address). Backend selection is priority-based (qdrant > weaviate > pgvector > lancedb) via VECTOR_DB_BACKEND/VECTOR_DB_URL; the vectordb block picks the matching backend through a factory (store_factory.rs), lancedb feature-gated.

The agreed tiering (operator decision):

Mosaic's current state.

• Deploy-lens engine sets (hard-validated): db: postgres | mysql | mariadb | redis | dynamodb | clickhouse, vector: opensearch | qdrant | weaviate | milvus | pinecone. No embedded options.
• The generated app does not connect to the deployed stores (the known MOSAIC_DB_URL/MOSAIC_VECTOR_URL gap): today a db:/vector: spec is store-workload proof, and the app persists to its own persistence: {backend: jsonl | sqlite} file store (rusqlite) — the sqlite tier is half-present already.
• The in-app vector store is the mosaic-knowledge crate (vendored into every generated app as knowledge-engine/): in-memory entries + BM25 keyword index + brute-force cosine over in-memory embeddings, RRF fusion, JSON-file persistence (Store::save/load). No backend seam, no ANN index, no lancedb.

The li7 on-prem deployment (ADR 0010/0020/0023) runs one release per external engine; a small-app tier (embedded sqlite + lancedb) has no representation there.

Decision

1. Two new engines, embedded, k8s target.

• db.engine: sqlite — the app's own file store; no db store workload. Valid only with persistence.backend: sqlite (fail closed: the plan errors otherwise — same hard-error discipline as the store fields).
• vector.engine: lancedb — embedded LanceDB in the app process; no vector store workload.

Both are k8s-only in this ADR (PVC-backed; aws/azure EBS variants are out of scope). For each embedded engine the deploy lens renders, on the app Deployment:

• a PersistentVolumeClaim + volume + mount (/data),
• env for the app:
  • sqlite: the app's existing persistence.path_env var set to /data/app.db (no new env name invented),
  • lancedb: MOSAIC_VECTOR_BACKEND=lancedb and MOSAIC_VECTOR_URL=/data/lancedb (path, never a URL).

2. Knowledge engine gains a vector-backend seam (the EE store_factory port). mosaic-knowledge Store vector operations move behind a backend trait with two impls:

• File — today's JSON persistence (default; apps that never opt in render and run byte-identical),
• Lance — feature-gated lancedb crate: one Lance dataset per collection, Arrow schema (id / content / embedding FixedSizeList<f32> / metadata JSON), HNSW index for ANN.

The generated app selects the backend at boot from MOSAIC_VECTOR_BACKEND (unset → File); the keyword (BM25) leg and the RRF fusion stay in Store and are backend-agnostic, so hybrid search is unchanged in shape — only the vector leg's storage/index differs.

3. app.scale: small | medium | large (opt-in; unset keeps today's byte-identical behavior — omitted db:/vector: fields still mean no store). When set, a deploy spec that omits db:/vector: gets tier defaults:

• small → db: sqlite, vector: lancedb
• medium → db: postgres, vector: lancedb
• large → db: postgres, vector: qdrant

An explicit db:/vector: on the spec always overrides the tier default (the spec name is the environment, per ADR 0010).

4. Staged outcome (documented, not hidden). medium/large defaults name external stores (postgres / qdrant) before the app can talk to them — the app-lens wiring of MOSAIC_DB_URL/MOSAIC_VECTOR_URL (postgres persistence backend, qdrant client) is a follow-up ADR. Until then the tier's external store deploys and is healthy; the app still uses its embedded store. small is fully end-to-end in this ADR.

Consequences

• tessera_yaml/plan: engine sets grow sqlite (db) and lancedb (vector); new app.scale field; per-cloud validation gains the embedded k8s-only rule; the sqlite↔persistence-backend consistency check.
• Deploy lens: embedded engines skip the store workloads and render the PVC/volume/env on the app Deployment instead (values.yaml gains an embedded: block). Conformance goldens for affected examples re-pin.
• mosaic-knowledge: lancedb optional dependency (feature-gated); Store vector path routed through the backend trait; the JSON File backend is the default and the public API of Store is unchanged.
• The li7 deployment gains a 6th release (li7-lite, app.scale: small): one app pod, no store workloads, PVC-backed sqlite + lancedb — the small-app tier proven end-to-end including AI (OpenRouter embeddings written into the Lance dataset, hybrid search over it).
• Out of scope (deliberate): app wiring to external postgres/qdrant (follow-up ADR), the EE CQRS hybrid mode (PostgreSQL event store + SQLite projection store, doc/cqrs/storage-backends.md Mode 3 — a separate follow-up ADR), aws/azure embedded volumes, lancedb server deployments (the product is embedded by design).