One Linux box, many services: Caddy, nginx, three gunicorns and a broker

How three production sites, on-demand TLS for customer domains, systemd workers and an MQTT broker share one machine without sharing failures.

2026 · infra

bytes uncompressed

123,345

the marketing page, now 23,904

backend tests

332

before the forty-column drop

browser assertions

255

the same destructive release

Context

The platform runs on one Linux machine — one box, which also hosts two other applications that have nothing to do with it, and whose PostgreSQL instance next to the code is the production database. There is no staging tier — the working tree that gets edited is the tree the application server reads.

The arrangement is cost-driven, and it was uneventful until June 2026, when every published store got a free {handle}.memberhub.live subdomain and paying stores got to point a domain they own at the platform instead. Certificates for an unbounded, database-driven set of hostnames are not something a pre-provisioned list can serve.

The constraint that decided the answer was not the platform's own, though. It was the two co-hosted applications: whatever terminated TLS for the new hostnames had to terminate it for the whole machine, because port 443 is not a thing you can half-own.

What I built

At the front is Caddy, holding ports 443 and 80 for the box and reverse-proxying plain HTTP to nginx on a loopback port. nginx keeps every server block it had — static files, media, the WebSocket upgrade for the real-time features, the marketing site as plain files — re-bound to that internal port with TLS stripped off. Behind it are three application services, one per site, each a gunicorn process group bound to a unix socket that systemd owns rather than the process. The socket therefore exists whether or not a worker does: the proxy can always connect, and a restart — the only option, since the unit supports no reload — never races a request.

1. Caddyholds ports 443 and 80 for the whole box
2. nginxon a loopback port, every server block kept, TLS stripped
3. systemd socketexists whether or not a worker does
4. gunicornone process group per site, three sites

Above the sockets, one database cluster serves all three applications, and one MQTT broker carries the platform's collaborative features — reached by browsers over the same origin and certificate through a proxied WebSocket path rather than a second open port. Two timers do the work that does not fit inside a request: one fires every five minutes and transcodes a pending video; one runs daily and rebuilds the marketing site's news feed, writing it atomically so a failure leaves yesterday's file in place rather than an empty one.

On-demand TLS mints and renews one certificate per hostname at that hostname's first HTTPS handshake. Left alone that is an invitation: anyone pointing a domain at the address could spend the issuing authority's whole weekly quota. The safety valve is an ask endpoint. Before issuing, Caddy calls the application with the hostname and gets back a 200 or a 403 from a view of about ten lines — allow the app and marketing hosts, then any host the store resolver recognises, otherwise refuse. What matters is which resolver it calls: the one the request middleware already uses to decide which store a request belongs to, so "which hosts get certificates" and "which hosts serve stores" cannot drift apart.

Host validation moved into that middleware wholesale. The framework's own allowlist accepts anything, with a comment naming the middleware as the authoritative check: known application hosts pass, everything else must resolve to a store or gets a 404. That closes Host-header injection and makes an unknown hostname cheap — one lookup and a refusal, cached per worker for sixty seconds and capped at ten thousand entries so probing cannot grow memory without bound.

Decisions

Front nginx, do not replace it. Moving four sites' worth of routing into a new server on the day I introduced automatic certificate issuance would have been two migrations at once. Layering left nginx untouched but for the port it listens on. The cost is an extra proxy hop, which is exactly where the trouble came from.

No ownership token for customer domains. The usual design asks the owner to publish a DNS TXT record proving control. I skipped it: the first HTTPS request that actually arrives for a pending hostname proves more than a TXT record does — the DNS points here, and issuance for it succeeded. The middleware flips the domain from pending to active on that first hit and serves the store.

Deploy as though the tree is dirty. Deploys copy changed static files by name rather than running the framework's collect step, because the tree routinely holds half-finished work of my own that a blanket copy would ship. Templates render from source, so editing one is a deployment in itself and is quarantined into the last step.

What went wrong

A redirect loop invisible from outside. With Caddy speaking plain HTTP to nginx, nginx saw an unencrypted request, so the application's HTTPS redirect fired on requests that had arrived over HTTPS. A forwarded-protocol map fixed it. The sequel was better: the ask endpoint has to set that header itself, and including the standard proxy block alongside sent it twice. The application read the first value, decided the request was insecure, and answered the ask with a redirect — which the certificate issuer will not follow. Certificates were refused for a reason that had nothing to do with the answer.

Compression that stopped at the middle proxy. Months later I measured page weights and found nothing compressed. nginx's gzip was configured and correct; it never fired, because the front proxy strips the client's encoding header on the way through, so nginx saw a client that had not asked for compression. Compression had to move to the front; the two layers are told apart by curling the internal port directly with a Host header. After the fix, one bundle went from 46,279 bytes to 15,798, another from 14,191 to 2,858, and the marketing page from 123,345 to 23,904.

Forty columns dropped on the production database. Retiring an old storefront renderer meant dropping around forty columns from a live settings table with no staging copy to rehearse on. The controls were written down before anything ran: dump the database; split it into a deploy that stops reading the columns and a second that drops them, so no running code references a column that is gone; put the data-rescue steps inside the drop migration ahead of the removals, so rescue and destruction cannot be separated; and send the owner a heads-up immediately before executing.

Outcome

Three applications, an unbounded set of storefront hostnames and a customer-owned domain share one machine, each with a self-renewing certificate and none able to serve another's data: a store's hostname is confined to storefront paths, and another organisation's public data on it is a 404. Dozens of deploys have gone out from this live box, including that forty-column drop, with no data-loss incident so far — the destructive one verified by 332 backend tests and 255 browser assertions.

The limits are written down rather than hidden. On-demand issuance spends one certificate per hostname against a weekly quota per registered domain — fine at dozens of stores, with an escape hatch planned: one wildcard certificate for the subdomains. One older renewal for the co-hosted sites still runs beside Caddy's; the platform's own port-80 renewals were disabled outright rather than left to fail quietly every night. And sharing a box means sharing failure domains: the day the co-hosted application's own hardening turned off anonymous access on the broker, the platform's real-time features stopped connecting without a line of its code having changed.