Deployment testing

How mosaic apps are tested against real clusters and real clouds (ADR 0010: one app, many targets).

The three workflows

All three use the same entry point:

scripts/e2e/harness.sh <helm-chart-dir> <app-image> [namespace] [extra helm args...]

The harness applies the peer fixtures (scripts/e2e/peers/: postgres CDC source, mysql poll source, clickhouse sink), seeds them, installs the chart, port-forwards the app, and asserts:

1. /health responds;
2. GET /api/sync lists all three mirrors;
3. backfill lands the 2 non-draft orders in ClickHouse (where clause honored) and the 3 snapshot order_items;
4. CDC: a new row inserted into the postgres peer appears in ClickHouse;
5. POST /api/sync/{mirror}/once is accepted;
6. poll→file: the JSONL archive contains the CRM customers;
7. poll→model: GET /api/order contains the derived customers.

On any failure it dumps pods, events, and the app's logs.

Running the k3d e2e locally (optional)

The CI job does it for you; to iterate locally (needs Docker + k3d):

curl -sSL https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=5.6.0 sh
k3d cluster create e2e --image docker.io/rancher/k3s:v1.30.5-k3s1

cargo run -q -p mosaic-cli -- build examples/data-sync -o /tmp/out-ds
(cd /tmp/out-ds && docker build -f deploy/e2e-k8s/Dockerfile -t mosaic/data-sync-e2e:local app/)
docker save mosaic/data-sync-e2e:local | k3d image import -c e2e -

bash scripts/e2e/harness.sh /tmp/out-ds/deploy/e2e-k8s/helm mosaic/data-sync-e2e:local mosaic-e2e --set image.pullPolicy=Never

k3d cluster delete e2e

Registering the on-prem runner

The on-prem workflow needs a GitHub Actions runner labeled mosaic-onprem with, on the box:

• docker (for the image build),
• kubectl configured against your k3s cluster (KUBECONFIG in the runner's env),
• helm,
• the k3s CLI (image import via k3s ctr images import),
• rustup (the render step) — or a prebuilt mosaic-cli.

Register:

curl -sSfL https://raw.githubusercontent.com/actions/runner/main/docs/scripts/add-self-hosted-runner.sh | bash -s <URL> <TOKEN> mosaic-onprem

Then dispatch e2e-onprem from the Actions tab. The harness reuses the mosaic-e2e namespace on your cluster — peers and the app are (re)installed there; the image is tagged mosaic/data-sync-e2e:local, so prune old ones with k3s ctr images rm as you like.

Deploying to the on-prem cluster

Public-app infrastructure

The public-app target is the single-node k3s cluster on li7 (192.168.11.63). It is the real on-prem hosting environment for public Mosaic applications, separate from the disposable k3d cluster used by e2e-k8s.yml on GitHub-hosted runners.

The network path is:

<hostname>.eisler-systems.de
  -> Netcup wildcard DNS / FRITZ!Box public address
  -> FRITZ!Box port forwarding (80/443)
  -> li7 k3s Traefik
  -> Kubernetes Ingress
  -> Mosaic Service

Use a public eisler-systems.de hostname for an application Ingress and set the class to Traefik. cert-manager is installed in the cluster and the letsencrypt-prod ClusterIssuer obtains and renews certificates through the Traefik HTTP-01 path. The deployment workflow therefore enables TLS and sets the host, secret name, and ClusterIssuer without storing certificates in the repository.

The persistent eugeis/mosaic self-hosted runner on li7 is labeled mosaic-onprem and has Docker, Rust, Helm, kubectl, and the k3s CLI. Its kubeconfig is local to the runner at /home/ee/.kube/config; workflows do not upload a kubeconfig secret to GitHub-hosted runners or expose the Kubernetes API publicly.

Dispatch deploy-onprem.yml to deploy a rendered k8s spec to the local k3s cluster. The workflow runs only on the mosaic-onprem self-hosted runner, which has the cluster kubeconfig installed locally. It does not use a GitHub kubeconfig secret or expose the Kubernetes API to GitHub-hosted runners.

The workflow renders the selected app, builds its generated Dockerfile, imports the image through k3s ctr images import, and runs helm upgrade --install. The default inputs deploy examples/orders using its helm deployment spec to the mosaic namespace at app.eisler-systems.de. The host input overrides the generated chart's ingress host; cert-manager and the cluster ingress controller remain responsible for TLS.

The self-hosted runner is trusted with cluster-admin access. Keep this workflow manual and restrict repository write access accordingly: a workflow executed on this runner can change the local cluster.

Cloud e2e (AWS / GCP)

The cloud job assumes the spec's Terraform was applied once (it creates the cluster, the registry, the managed stores, the namespace, and the DSN secrets the chart reads). Per cloud:

AWS (deploy/prod-aws/)

cd examples/full-stack   # after: cargo run -q -p mosaic-cli -- build examples/full-stack -o /tmp/out-fs
cd /tmp/out-fs/deploy/prod-aws/infra
terraform init && terraform apply     # EKS + ECR + RDS + OpenSearch + secrets

Repo configuration (Settings → Secrets and variables → Actions):

The job builds the image, pushes it to ECR, helm installs deploy/prod-aws/helm (db + vector mode: external — the DSN secrets come from Terraform), waits for readiness, and smokes /health plus a write through the managed RDS event store, then uninstalls.

GCP (deploy/prod-gcp/)

cd /tmp/out-fs/deploy/prod-gcp/infra
terraform init && terraform apply     # GKE + Artifact Registry + Cloud SQL + db secret

Note: GCP has no first-party managed Qdrant with a stable Terraform resource, so prod-gcp runs qdrant in-cluster on GKE (managed: false); only the postgres store is managed (Cloud SQL). The plan layer rejects managed combinations without a stable URL resource (deploy-managed-unavailable) — see the matrix in ADR 0010.

Azure / Alibaba

azure-aks and alibaba-ack generate the same infra/ + helm/ + Dockerfile triplet (AKS module + ACR; ACK module + ApsaraDB RDS / Redis / ClickHouse). There is no automated cloud job for them yet — run the AWS job manually as a template: apply the terraform, build/push the image from deploy/<spec>/Dockerfile, helm install deploy/<spec>/helm, smoke, uninstall.

Cost notes

• e2e-k8s (k3d) runs on a shared runner: a few minutes of CPU, no bill.
• The on-prem run uses your existing cluster.
• Cloud jobs are manual only and only touch resources you provisioned; the job uninstalls the Helm release but never destroys Terraform state. terraform destroy when you are done.

Troubleshooting

• App pod stuck ContainerCreating on a cloud target: the DSN secret is missing — kubectl -n <ns> get secret <app>-db <app>-vector. They are created by Terraform, not the chart.
• ImagePullBackOff on k3d: the image import step failed or the tag drifted — the caller passes --set image.pullPolicy=Never precisely so a preloaded image wins.
• CDC never lands: the peer postgres must run wal_level=logical (the fixture sets it); check the app logs for replication errors.