Bi-directional KV transfer cuts vLLM time to first token up to 3.3x

AWS published benchmarks showing that a new way of moving saved work between servers makes AI agents start answering faster as conversations grow. In tests, time to first token, the wait before the first word of a reply appears, fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8. The feature is built into vLLM, a popular open-source tool for running AI models.

Why long chats slow down

Agents work in turns. Each turn, the app resends the whole history: instructions, earlier questions, tool results and past answers. The model must process all of it again. Some setups split the job across two kinds of servers. A prefill server reads the prompt and builds a memory of it, called the KV cache. A decode server takes that memory and writes the reply.

Normally the memory only travels one way, from prefill to decode. So the prefill server never gets back the memory of the reply it just helped produce. It has to redo that work next turn. If its cache was cleared, it redoes the entire conversation.

What changed

Bi-directional KV transfer lets the decode server send its saved memory back to the prefill server on later turns. The prefill server then skips recomputing text it has already seen. A setting called the KV recompute threshold controls when a transfer is worth doing. A lightweight router tracks where each conversation's saved blocks live, keyed by a conversation_id. It stores locations only, not the memory itself.

• Decode server keeps eligible memory after it replies.
• It sends location details back in its streaming response.
• The router passes those details to the next prefill request.
• Prefill pulls the memory over, then computes only what is new.

The numbers

AWS tested a setup with one prefill and one decode server on Amazon EC2 P5en instances, linked by AWS EFA, a fast network for moving data between servers. Qwen3-32B saw up to 1.6x lower time to first token. GLM-5-FP8 saw up to 3.3x. AWS says results vary with workload and configuration, so those are best-case figures, not guarantees.

Who it affects

This matters most to teams that host their own models for chat or voice agents, where a slow first word is noticeable. AWS says it first built and tested the feature on its Trainium chips, then contributed it to vLLM as a capability that is not tied to one type of chip. The new tests show it also helps on GPUs. Teams using hosted AI services from a vendor will not set this up themselves.

What is not known yet

The material gives no pricing and no release date. vLLM includes the connector-side support, but a router that tracks conversations is also needed. vLLM's GitHub has an example proxy that can be adapted. Results on other hardware, larger setups or different workloads were not reported.