As enterprise demand for Large Language Model inference scales, cloud operators face the challenge of maximizing GPU utilization while maintaining low latency. Traditional static GPU allocation leads to severe resource underutilization, while standard scheduling algorithms suffer from head-of-line blocking due to the unpredictable length of model outputs.

This talk introduces a dynamic, open-source infrastructure blueprint that addresses these challenges by combining the latest OpenStack accelerator management capabilities with advanced scheduling theory.

We demonstrate how the revitalized OpenStack Cyborg project, in coordination with Nova and Ironic, enables dynamic, multi-tenant GPU provisioning and lifecycle management. By leveraging Cyborg drivers, operators can dynamically partition and attach physical accelerators to virtual machines or bare-metal nodes based on real-time demand.

To optimize the software layer above this hardware foundation, we integrate an uncertainty-aware scheduling framework inspired by recent academic research. Rather than relying on static predictions of output lengths, we implement a Tail Inflated Expectation scheduler. This approach models output length as a heavy-tailed distribution, mitigating the risk of long-running requests blocking shorter requests.

Attendees will learn how to deploy this integrated stack to achieve high throughput and low per-token latency in sovereign AI clouds.