Continuous Batching (Iteration-Level Scheduling)

What is Continuous Batching?

Continuous batching, also called iteration-level scheduling or in-flight batching, is an inference technique that schedules and evicts individual requests at the granularity of a single token generation step, rather than at the level of a whole batch. It removes much of the GPU idle time that traditional batching strategies leave on the table when serving large language models.

Large language models generate text autoregressively: they produce one token at a time, conditioned on all preceding tokens. That process is sequential by nature, but serving these models at scale requires processing many requests at once to keep expensive GPUs utilized.

To see why continuous batching helps, it's useful to look at how LLM text generation actually runs. Generation happens in iterations. Each iteration produces exactly one new token for every sequence currently being processed by the model.

In traditional systems, a batch of requests is grouped together and processed until the longest request in the batch is fully generated. Because request lengths and generated output lengths vary widely in real-world applications, shorter requests finish early, but they cannot leave the batch until the longest request concludes. The GPU ends up computing "padding" tokens, wasting compute cycles and memory bandwidth on requests that have already finished.

Continuous batching solves this by decoupling the batch from the lifecycle of any individual request. First introduced in the Orca paper (OSDI 2022), continuous batching evaluates the state of the batch after every single token iteration.

When a request generates its final token (an end-of-sequence token, or it hits a max token limit), it is immediately evicted from the batch. The system then pulls a new pending request from the queue and inserts it into the newly freed slot for the next token iteration. This keeps the GPU saturated with productive work. Anyscale's benchmarks of Orca-style scheduling reported throughput improvements of up to roughly 23x compared to naive batching, an order-of-magnitude gain in many production workloads.

Static Batching vs. Dynamic Batching vs. Continuous Batching

Batching techniques have evolved to squeeze more throughput out of AI accelerators. Here is how continuous batching compares to its predecessors.

Static Batching requires padding all prompts and generated sequences to a predetermined maximum length. It is rigid, wastes large amounts of memory, and is largely obsolete for LLM serving today.

Dynamic Batching improves on static batching by only padding up to the longest sequence in the current batch. However, it still suffers from the "convoy effect": if one request generates 1,000 tokens and another generates 10, the short request ties up memory and compute resources for 990 iterations doing nothing.

Continuous Batching operates fluidly. The batch size stays roughly constant, but the composition of the batch changes at every step. There is no convoy effect and no wasted padding computation.

PagedAttention: Eliminating GPU Memory Fragmentation in the KV Cache

While continuous batching solves the compute inefficiency problem, it exposes another bottleneck: GPU memory fragmentation.

During autoregressive generation, LLMs cache the Key and Value (KV) vectors for past tokens to avoid recomputing them. This KV cache grows as new tokens are generated. Early continuous batching systems had to pre-allocate contiguous chunks of GPU memory for the maximum possible length of a request, to guarantee it wouldn't run out of memory mid-generation.

Because most requests never reach the maximum length, this contiguous pre-allocation caused severe internal fragmentation (wasted pre-allocated space) and external fragmentation (unusable gaps between memory chunks). Often, over 60% of KV cache memory was wasted, limiting the number of concurrent requests the GPU could hold.

The solution came with PagedAttention, introduced by the researchers behind vLLM. Inspired by operating system virtual memory and paging, PagedAttention breaks the KV cache into fixed-size blocks (for example, blocks that hold 16 tokens each).

Instead of pre-allocating memory, PagedAttention allocates these blocks on demand as the sequence grows. The blocks do not need to be contiguous in physical GPU memory. A block table maps the logical sequence of tokens to their physical blocks, similar to an OS page table.

By eliminating memory fragmentation, PagedAttention lets the system pack more requests into the GPU's VRAM. Larger batch sizes mean higher throughput, making PagedAttention a common companion to continuous batching.

Chunked Prefill and Decode Phase Disparities

LLM inference has two distinct phases with very different hardware utilization profiles:

1. The Prefill Phase: When a new request arrives, the engine must process the entire prompt to generate the first token. This phase is highly parallelizable, relies heavily on matrix multiplications, and is typically compute-bound.
2. The Decode Phase: After the first token, the engine generates subsequent tokens one by one. This phase reads the KV cache and network weights for every single token, making it memory-bandwidth-bound.

In a continuous batching system, the batch is often a mix of requests: some in the prefill phase (newly inserted), others in the decode phase. Injecting a large prompt (for example, 32,000 tokens) into the prefill phase can cause a latency spike for all the decoding requests in the same batch, since the GPU gets bogged down processing the large prompt.

To mitigate this, modern inference engines use Chunked Prefill. Instead of processing a large prompt all at once, the engine splits the prompt into smaller chunks (for example, 512 tokens at a time), and schedules these prefill chunks alongside decode iterations. This balances the workload, keeping both the compute cores (used by prefill) and the memory bandwidth (used by decode) busy without causing jitter or latency spikes for users waiting on their next token. In vLLM's V1 engine (the default since 2025), chunked prefill is on by default: the scheduler works from a single token budget per iteration (max_num_batched_tokens) rather than distinguishing prefill requests from decode requests at all.

Prefill-Decode Disaggregation

The largest deployments now take phase separation one step further: instead of interleaving prefill and decode on the same GPUs, they run the two phases on separate GPU pools and ship the KV cache from prefill workers to decode workers over the network.

This is called prefill-decode (P/D) disaggregation. Because prefill is compute-bound and decode is memory-bandwidth-bound, dedicating different hardware to each phase lets operators scale, batch, and even parallelize them independently, and it removes the interference between long prompt processing and steady token streaming entirely. The pattern was popularized by Moonshot AI's Mooncake architecture (which serves Kimi) and DeepSeek's inference clusters, and NVIDIA built its Dynamo serving framework (announced at GTC in March 2025) around it. As of 2026, vLLM, SGLang, TensorRT-LLM, and Dynamo all support disaggregated serving, with libraries such as NVIDIA's NIXL and Mooncake's transfer engine handling KV cache movement between nodes over RDMA.

The tradeoff is operational complexity and network cost: transferring multi-gigabyte KV caches between pools requires fast interconnects, so disaggregation pays off at large scale and long context lengths, while single-node deployments are better served by chunked prefill within one continuous batch.

Inference Engine Landscape: vLLM, TensorRT-LLM, TGI, and SGLang

The rapid adoption of continuous batching has produced a competitive landscape of open-source and proprietary inference engines.

• vLLM: Developed at UC Berkeley, vLLM popularized PagedAttention and open-source continuous batching. It is one of the most widely used community engines today, known for its ease of use, broad model support, and strong baseline throughput.
• TensorRT-LLM (TRT-LLM): NVIDIA's official, highly optimized inference library. It uses what NVIDIA calls "in-flight batching" (their term for continuous batching). TRT-LLM is tuned closely to NVIDIA hardware, using custom kernels, but has a steeper learning curve than vLLM.
• Text Generation Inference (TGI): Built by Hugging Face, TGI was one of the earliest adopters of continuous batching in production. It powers Hugging Face's inference endpoints and offers robust features for production deployments, including deep integrations with the Hugging Face Hub.
• SGLang: A newer engine that builds on the concepts of vLLM but introduces RadixAttention. SGLang optimizes continuous batching for complex, structured generation tasks (like JSON output or multi-turn chat) by automatically sharing the KV cache across multiple requests that share identical prompt prefixes. It also complements techniques like speculative decoding and context caching for further throughput gains.

Continuous batching sits alongside other serving optimizations covered elsewhere in this glossary, including speculative decoding for reducing per-token latency and context caching for reusing KV cache state across requests with shared prefixes. The underlying architecture these engines serve is almost always a transformer.

Frequently Asked Questions

What metrics matter when tuning a continuous batching deployment? Four numbers cover most tuning work: Time To First Token (TTFT, dominated by queueing and prefill), Time Per Output Token (TPOT, also called inter-token latency, dominated by decode batch size), total throughput in tokens per second, and KV cache utilization. The core dial is the per-iteration token budget (max_num_batched_tokens in vLLM): raising it improves throughput at the cost of TPOT jitter, lowering it does the reverse. Many teams now optimize for "goodput," the throughput of requests that meet their TTFT and TPOT service targets, rather than raw tokens per second.

Does continuous batching increase latency? No, in most cases it decreases average latency. Time To First Token (TTFT) might see a small increase under maximum load due to queueing, but inter-token latency and overall end-to-end latency drop significantly because the GPU is not wasting time computing padding tokens.

What is the difference between Continuous Batching and In-Flight Batching? There is no difference. "In-Flight Batching" is the terminology NVIDIA uses in their TensorRT-LLM documentation to describe iteration-level scheduling. The underlying mechanism, evicting finished sequences and injecting new ones per iteration, is identical.

How does continuous batching relate to prefix caching? They are complementary but distinct optimizations. Continuous batching optimizes the scheduling of requests, so no cycles are wasted on finished sequences. Prefix caching optimizes memory and prefill compute by reusing the KV cache for prompts that share identical beginnings, such as a system prompt. Modern engines use both simultaneously to maximize throughput.

Can I use continuous batching on consumer GPUs? Yes. Inference engines like vLLM run well on consumer hardware, including current-generation cards. The primary constraint on consumer GPUs is VRAM; continuous batching paired with PagedAttention ensures that the limited VRAM on these cards is used as efficiently as possible, allowing for solid local throughput.