setloop.io
White Paper · SET-WP-2026-04
All papers
setloop.io
White Paper
Setloop Technical White Paper Series

Improving AI Compute Economics through GPU Workload Optimisation

A practical approach to improving AI inference and training through workload-aware scheduling, dynamic batching, KV-cache management and cost attribution.

Document
SET-WP-2026-04
Published
September 2026
Version
1.0
Category
Performance Engineering
Classification
Public
Working infrastructure, not slide decks. setloop.io

In brief

The challenge. Poorly configured GPU infrastructure raises the total cost of ownership (TCO) of enterprise AI. As organisations move generative AI workloads into production, idle capacity and inefficient serving become material operating costs.

The response. Better AI economics depend on more than procuring hardware. Scheduling, dynamic batching and KV-cache management determine how effectively each accelerator is used.

What this paper covers. This paper examines the economics of inefficient inference, the principal optimisation patterns, and the engineering needed to apply them in production.

Measure
Utilisation, throughput, latency and cost against a defined baseline
3 patterns
Scheduling, dynamic batching and KV-cache management
TCO ↓
Lower total cost of ownership by recovering idle capacity
01

The economics of inefficient inference

Buying GPU capacity does not ensure that it will be used efficiently. Without a workload-aware architecture, organisations may pay for idle hardware while throughput and latency remain constrained.

The utilisation gap is widespread. In the 2024 State of AI Infrastructure at Scale survey, conducted by ClearML with the AI Infrastructure Alliance and FuriosaAI, roughly two-thirds of organisations reported that no more than 70% of their purchased GPUs were in use, even at peak demand. Only 7% reported utilisation above 85%. The IEA projects that global data-centre electricity consumption will rise from about 415 TWh in 2024 to roughly 945 TWh by 2030, with AI-driven accelerated servers contributing almost half of the increase. Idle GPU capacity therefore carries both a financial and an energy cost.

2–4x
Higher LLM serving throughput at equivalent latency demonstrated by vLLM's PagedAttention versus prior state-of-the-art systems (Source: Kwon et al., SOSP 2023)
7%
Share of organisations achieving more than 85% GPU utilisation at peak demand (Source: ClearML / AI Infrastructure Alliance / FuriosaAI, 2024)
945 TWh
Projected global data center electricity consumption in 2030, roughly double 2024 (Source: IEA, Energy and AI, 2025)

Memory bottlenecks

Naive inference architectures manage key-value memory inefficiently, constraining throughput and increasing latency.

Suboptimal batching

Static batching limits the number of concurrent requests an accelerator can process efficiently.

Opaque spending

Scaling inefficient workloads raises cloud costs when teams cannot attribute model spending to products, features or customers.

02

Core optimisation architecture

The economics of AI inference depend on how the serving system schedules requests, batches work and manages memory. The following patterns address those constraints.

Advanced scheduling algorithms

Route and prioritise requests according to service objectives, available capacity and workload characteristics.

Dynamic batching

Group variable-length inference requests at runtime to use accelerator capacity more efficiently.

KV-cache optimizations

Manage key-value memory using techniques described in the vLLM paper (Kwon et al., 2023) and NVIDIA's TensorRT-LLM optimisation guidance.

The effect of these changes must be measured against the existing workload. Relevant measures include utilisation, throughput, time to first token, inter-token latency, queueing delay and cost per successful request. Improvement will vary by model, traffic pattern, service objective and hardware configuration.

High-bandwidth interconnect does not remove software bottlenecks. Fifth-generation NVLink provides 1.8 TB/s of bidirectional bandwidth per GPU, while NVSwitch extends a coherent NVLink domain across 72 GPUs in NVIDIA's GB200 NVL72, with 130 TB/s of aggregate bandwidth. Depending on the workload, throughput may still be limited by memory management, batching, scheduling or communication. Optimisation should begin with measurement rather than an assumption about the dominant constraint.

03

How Setloop can help

Applying research techniques in production requires expertise in model serving, distributed systems, performance measurement and cost engineering. Setloop combines these disciplines in architecture reviews and implementation work.

Architecture reviews

Setloop reviews training, fine-tuning, inference, RAG, agent, batch and other GPU-intensive workloads.

Working infrastructure over slide decks

Setloop can take recommendations through benchmarking, implementation and deployment in the customer's environment.

Integrated FinOps

Setloop links performance engineering with AI FinOps, including token-level cost tracking, provider reconciliation and feature-level attribution.

Key takeaways

What infrastructure leaders should remember

  1. Utilisation is an engineering outcome. More GPU capacity will not correct inefficient scheduling, batching or memory management.
  2. Measure the actual constraint. KV-cache efficiency often affects inference throughput and latency, but the dominant bottleneck depends on the model, traffic and hardware.
  3. Batching changes the economics. Dynamic batching can improve parallelism, but it must be tuned against latency objectives and request variability.
  4. Connect optimisation with FinOps. Token-level cost tracking, provider reconciliation and feature attribution link model spending to product value.
05

References

  1. Kwon, W. et al. (2023). “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of SOSP (the vLLM paper).
  2. NVIDIA. “TensorRT-LLM Optimization Guides.” NVIDIA developer documentation.
  3. NVIDIA. “NVLink and NVSwitch: fifth-generation specifications” and “GB200 NVL72.” NVIDIA product documentation.
  4. ClearML, AI Infrastructure Alliance, and FuriosaAI. “The State of AI Infrastructure at Scale 2024.” March 2024.
  5. Stanford HAI. “AI Index Report 2025.” Stanford University, April 2025.
  6. International Energy Agency. “Energy and AI.” April 2025.

About Setloop

Setloop is an engineering consultancy and product studio for organisations building GPU workloads, cloud GPU platforms, private AI infrastructure and AI factory architectures. Its engineers design, benchmark and deploy operational infrastructure in customers' private and hybrid-cloud environments across the UK and EU.

The Setloop Technical White Paper Series distils reference architectures from client work and the Setloop product portfolio, including LLMTrace, AutoOps, GPU Cloud Platform, AI FinOps and Automatic RL Research.

For more information