On 10 September the staging cluster came up: three Talos control plane nodes and two system nodes on Scaleway Instances, every one of them declared in OpenTofu, and a cluster that could schedule a pod the moment etcd was bootstrapped. Until then the estate existed as a runbook and whatever I had clicked in the console.

Why not the managed offer#

Scaleway sells a managed Kubernetes, and the obvious question is why I did not use it. The platform will run images its customers supply, so a namespace is not a security boundary and the kernel has to be one. That means a sandboxed runtime (gVisor, with Kata for what gVisor cannot host) on shared nodes. The managed node pools accept only the stock container runtimes, and their node images are not mine to extend. The managed control plane also offers no audit policy of mine, no say over the API server's flags or encryption at rest, and a capped etcd, while every customer's Helm history would live in that etcd.

So the control plane is mine: three nodes, for quorum rather than for cost. Staging runs the shape production will run at the smallest size that still proves it, because a quorum that only exists in production is a quorum nobody has rehearsed losing. Talos on every node means one operating system everywhere, and so one image pipeline and one upgrade mechanism. The design, costed before any of this was built, came to about 218 euros a month before the first tenant.

Three layers, because of when providers are configured#

The OpenTofu layout came first, the day before, and it creates nothing. Two decisions in it are forced rather than stylistic, and both are expensive to discover after state exists.

Each environment is a directory rather than a workspace. tofu workspace select keeps whatever was selected last, silently, so a workspace layout offers a way to reach production by forgetting a command rather than by typing one.

Each environment is also split into roots: one for the cloud resources, one for the cluster built on them. Provider configuration is evaluated at plan time, so a single root that creates a cluster and then configures a Kubernetes provider from its own outputs cannot plan even once. The cluster root reads the cloud root's state instead.

OpenTofu rather than Terraform was forced too. On S3-compatible storage the backend's lock object ignored a checksum flag, so state writes succeeded while lock acquisition failed; OpenTofu fixed it, and the equivalent Terraform report closed as a duplicate with a workaround. OpenTofu also encrypts state, which the Terraform CLI does not do at all. The encryption block carries no passphrase, so formatting, linting and validation all pass on a checkout with no secret in it, and CI has no credential to apply with. Adding one would be a visible change to a workflow rather than a quiet change of behaviour.

The linter paid for itself on day one: it found the cluster root pinning two providers that nothing used, which cost a cold run 158 MB of downloads instead of 43.

The cloud layer#

One private network with a shared gateway, a PostgreSQL instance reachable only from inside it, the application's bucket, a registry namespace, secret containers, and the five machines. Sizes and counts are variables, so a second environment is a variables file rather than a fork.

The Scaleway API turned three assumptions over. The address I had reserved for the edge belonged to a family no load balancer can bind, and the two families do not convert. The credential that builds the estate holds no IAM permission, which is the right answer anyway: writing IAM needs a grant whose holder can grant itself anything, so those applications stay manual. And the SDK prefers its own environment variables over the arguments the provider passes, so with the usual ones exported the shell, not the variables file, decided which project a root built in. They are renamed now.

One detail is worth reading twice. The database password goes into Secret Manager as a write-only attribute, which keeps it out of state and plan files. A write-only value is only re-sent when its version counter changes, and it reads back as nothing. With the counter pinned to a constant, a new password would reach the database while Secret Manager kept the old one, and nothing would fail. One input now both generates the password and versions the secret, so a rotation moves both halves in the same apply.

The cluster layer#

The cluster root writes the machine configuration for both roles, applies it to machines that have joined nothing yet, bootstraps etcd, and carries Cilium, the cloud controller manager and the storage driver in as manifests, so the cluster comes up already able to run a workload.

The nodes have no public address. The Kubernetes API sits behind a private load balancer rather than a Talos virtual address, which is the cheaper answer and the one I wrote first. Scaleway wants a MAC registration before it carries an address it did not allocate, rate limits the broadcast a failover announces itself with, and says nothing about whether it forwards frames for an address it never assigned. Two unknowns for one address was the argument against it.

Reaching a node's own Talos API is a switch that is off at rest. A node in maintenance mode, which is exactly what a new or wiped node is, presents no certificate and accepts a configuration from whoever asks, so that door opens only while I am using it.

The machine secrets bundle, the cluster's identity, is imported rather than generated, and OpenTofu refuses to destroy it. A second bundle is not a repair; it is a different cluster the running nodes will not talk to, and getting it wrong means resetting all five machines.

The afternoons#

Four settings each encode a trap that took an afternoon to find rather than to fix.

The ephemeral volume is selected by size rather than by device path, because the path in the Scaleway guide is right until volumes are reattached in another order, and then it names the system disk. The gateway's address has to be in the machine certificates, and that omission is the expensive one: the provider probes a machine in maintenance mode without verifying anything, so the first apply passes and the refusal arrives later from a machine that can no longer be reached to fix it. The installer image is pinned to the one carrying the sandboxing extensions, because the stock installer would upgrade gVisor and Kata off every node and report success. And the kubelet is bound to the private subnet, or it may register an IPv6 address beside the IPv4 and every check that looks a node up by its IPv4 reports a node it cannot find.

The apply mode is a variable because the safe setting depends on the operation. The aggressive one, on a running cluster, reboots every control plane node at once, since nothing orders the applies, and takes etcd quorum with it.

Two things showed up only once charts were rendered into machine configuration. Hubble's chart mints a certificate authority inside the template, so two renders of the same version differ and the configuration never settles; it is off. And Talos ships with discovery enabled against a hosted service, so a cluster with no public address would still announce its membership to a third party. That is off too.

Where it went next#

A Talos node needs a custom image per environment and per upgrade, and two of the six manual commands that built one failed late: the storage import refused a raw image only after a 4.45 GB upload. On 27 September they became one script that reads the image header before uploading anything, reuses what an earlier run left, and refuses an image a wait left in error, because the CLI's waits exit successfully on failure as well as on success.

Everything after the cluster, from secrets to certificates to the application itself, is delivered from git by Argo CD rather than by OpenTofu. That is a post of its own.

Bootstrapping a self-hosted Kubernetes estate on Talos and Scaleway