This note covers the operations side of Petrichor Labs: how we isolate tenants on the network, how we test failover, how we triage CVEs, and how monitoring and paging are set up. Petrichor sells hosting that stays outside US jurisdiction, so we can't lean on a hyperscaler's managed services for any of this.
Why we run it ourselves
The managed pieces a cloud team would normally reach for, like load balancing, DNS and observability, mostly come from US-controlled providers. Using them would bring back the dependency our customers are paying to avoid. So each of those has to be something we run on our own hardware.
The base layer is bare metal provisioned with Terraform. Kubernetes runs on top of it, using k3s and Talos. Everything below sits on that.
Tenant isolation with a private mesh
Each tenant's workloads run on an encrypted overlay network built with Tailscale clients and a Headscale coordination server that we host ourselves, so the mesh doesn't depend on Tailscale's hosted control plane. A tenant's services see one flat private network no matter which physical hosts they're scheduled on, and they can't see anything that belongs to another tenant.
I chose this over firewall rules as the main isolation mechanism because mesh membership is tied to a key. A node either holds a key for that tenant's mesh or it can't reach that mesh at all. We still have firewall rules, but one bad change to a rule doesn't open a path between two tenants by itself.
It also means traffic between a tenant's services is encrypted on every hop, including across our own switches.
Failover drills
For each service we document where the standby lives, what replicates synchronously and what replicates asynchronously, and what gets promoted and how. That design work matters, but I don't count a failover path as working until we've run it against production-shaped load.
So we run a failover drill every month. For each one we record when it ran, who watched it, how long promotion took, and anything that went wrong. One drill turned up a problem with how DNS switched traffic over to the standby.
CVE triage
We're a small team, so we can't patch every advisory the day it's published and still ship anything. Each new CVE that touches our stack goes through three questions:
- Is the affected component internet-facing?
- Is the vulnerable code path reachable in our configuration?
- Is it being actively exploited?
A yes to any of them puts it on the urgent path. Everything else goes into the weekly patch cycle.
CVE-2026-31431, the Linux kernel bug known as "Copy Fail", is a good example. It's a local privilege escalation in the kernel's algif_aead crypto interface, so it isn't internet-facing. But on hosts shared by several tenants, a local user getting root is exactly what isolation is supposed to stop, and CISA listed it as actively exploited in May 2026. That put it on the urgent path for our kernels.
On a small team this tends to go wrong in one of two ways. Either every advisory becomes an interrupt and nothing else gets done, or everything waits for a monthly batch and a reachable remote code execution bug stays open for weeks. Having the questions written down means we've already decided which bucket an advisory goes in before it arrives.
Monitoring and paging
Monitoring runs on Prometheus and Grafana for metrics and dashboards, Alertmanager for routing pages, Zabbix for host and hardware checks, and Sentry for application errors.
The rule for paging is that every page has to need a person right now. Pages fire on symptoms a tenant would notice: failed requests, high latency, a service that's down. Hardware and host metrics go to dashboards we look at during working hours.
On bare metal it's tempting to alert on every disk, fan and NIC, because for once we can see all of it. I went the other way. Finding the cause usually happens after the page, in Grafana, so the page itself only needs to say what users are experiencing. We cut about 20 alert rules, and we now get around 15 pages a week.