Everyone wants LLMs — but for regulated organizations in the EU, sending every prompt to a US API is often a non-starter: DSGVO, data residency, sovereignty. We built the alternative: a self-hosted AI layer, assembled entirely from open-source, cloud-native components, where sensitive data never leaves our own infrastructure.

This is a hands-on account of a platform we designed, built, and run ourselves in-house — a practitioner's report, not a product pitch. We serve an open-weight model with vLLM, behind a model-agnostic gateway, exposed through open frontends for chat and coding — all on Kubernetes, with GitOps, mTLS, and per-user budgets and logging.

The core idea: sovereignty doesn't have to be all-or-nothing. The gateway is the control point. Sensitive workloads stay on our own model, while non-critical ones can be routed to frontier models — and we can add or swap any model without touching the applications, keeping governance, authentication, and cost control firmly in our hands.