By 19 September staging served the public site and the application over a certificate browsers trust, and no pipeline held a credential for the cluster. Past the cluster's own bootstrap, OpenTofu installs Argo CD and one root Application, and stops there. Everything else is a file committed under one directory, so what runs is a function of what is committed.

Nine days earlier the cluster could schedule a pod and do nothing else. This is how the space between the two was filled, in the order it had to be.

A third root, and where OpenTofu stops#

Argo CD needs Helm and Kubernetes providers configured from the kubeconfig the cluster layer produces. Provider configuration is evaluated at plan time, so it could not live in the root that creates the cluster: it got a third root of its own, which reads the cluster's state.

That root installs Argo CD and a single root Application pointed at the environment's directory in git. Every later component is a child Application committed there. I held that line on purpose, because each component added to OpenTofu instead would be one more thing nobody corrects when it drifts.

The root Application goes through a small Helm chart rather than OpenTofu's generic manifest resource, because that resource dry-runs against the server at plan time, and the Application's custom resource definition does not exist until the same apply installs Argo CD.

The webhook secret goes in with the install. Argo CD's webhook handler validates no signature when it has no secret, and adding the secret later would leave that open on the day a route appeared. The Argo CD server itself still has no public route: it would let the edge read the repository credentials in its namespace. The default project is narrowed to this repository, so an Application meant for the tenants' fenced project cannot land in the default one and slip past the fence. Prune and self-heal start off, and an Application gets them only once its tree has been seen to render the live cluster identically, or, like the site's, when it is alone in its namespace.

Secrets that never touch state#

A workload's credential had one path into the cluster: an OpenTofu root planting it, which puts secret material in state and makes every rotation an apply. External Secrets now reads Scaleway's Secret Manager through one cluster-wide store, and Reloader restarts whatever consumes a Secret that changed.

A few secrets cannot arrive that way, because they have to exist before External Secrets does: the key Argo CD reads git with, for instance. The secrets runbook names each of those exceptions, so a new one has to be argued rather than added.

To prove the whole path rather than trust it, a canary deployment logs the digest of a secret that guards nothing. Bumping that secret's version in Secret Manager has to change the digest the running pod logs, with nothing that matters touched.

One wildcard certificate, and a registrar the cluster cannot touch#

A wildcard certificate needs the DNS-01 challenge, and cert-manager has no solver for the registrar that holds the zone. The webhook alternative would have handed the cluster a token able to rewrite a zone that also carries the company's mail. Instead the challenge is delegated to acme-dns running in the cluster, which confines the cluster to the TXT records of one registration. Registration is switched off, its update API is reachable from cert-manager's namespace alone, and only DNS itself is public.

Staging is parked when it is not in use, torn down to a floor that survives, and that shaped the certificate work. The acme-dns registration is generated once and kept in Secret Manager, and an init container seeds it into a throwaway database on every start, so a bring-up never needs an edit at the registrar. Both Let's Encrypt endpoints are declared, and a new environment starts on the staging one: a park destroys the certificate, and the production endpoint allows five certificates a week for one set of names. Staging moved to the trusted endpoint once the untrusted chain had been issued and served for the host and its wildcard, and the certificate expiry watch runs daily from then on.

At the edge, Traefik's own resource definitions stay off, so nothing can declare a default certificate the usual way. An Ingress in the edge's namespace names the certificate's Secret instead. Traefik loads it into its one store before resolving any backend and picks from that store by server name, so every Ingress is served over it with no Secret copied anywhere.

Both origins, each from its own chart#

The public site and the application are separate origins with separate needs, so each has a chart. The site is an nginx image behind a Service and an Ingress: two replicas that roll without dropping below two, a read-only root filesystem, and a network policy that admits the edge alone.

Each chart refuses to render an image with no digest, or with a tag that does not name the environment. The web images bake their environment's origins into the build, so a floating or foreign tag would serve another environment's hosts with nothing reporting it. Before this, no gate read a chart at all. Now CI lints each one and diffs its render for each environment against a committed golden file, and a test holds each chart Application to the render that gate checked.

The application brought one more constraint. The edge can read every Secret in a namespace it serves, and the application's namespace holds the database password and its signing and encryption keys. So the workloads live in a namespace the edge does not serve, and the routes live in a second one holding no Secret, pointing across with ExternalName Services. The configuration scanner flags every ExternalName Service, so the exception is scoped to that chart's renders alone, and one anywhere else still fails. The gitops test refuses a synchronised secret in any served namespace other than a registry pull key, so the split cannot erode quietly.

Migrations had already moved out of the application's boot. A hook Job provisions the database on every sync, after the Secret and Redis and before the Deployments, so a failed migration stops the sync with the old pods still serving.

What the first sync found#

Two things failed on the first day, and neither was visible before a real browser or a real database met them.

The provisioning Job could not connect: the user that owns the managed PostgreSQL instance is not a superuser, and a database created through Scaleway's API revokes CONNECT from everyone, so the instance's own user could not open the one database it exists for. An explicit grant fixed it.

Every upload failed too. The browser uploads straight to a presigned URL, so the request is cross-origin, and the bucket refused its preflight because it had no CORS rule. The rule now admits exactly that request from the application's origin. That origin is the third copy of a value the web image bakes and the chart names, and a mismatch between them shows up only as a refused preflight in a browser, so a test now holds all three to the same build input.

What came next#

A monitor cannot report its own death, so telemetry and alerting were the next piece, placed outside the cluster on purpose, with the record kept outside its region. That landed on 27 September, and it is its own story.

From a bare cluster to a staging environment reconciled from git