Google and partners release free software to run AI models faster

Google and partners including Red Hat and IBM have released llm-d 1.0, free open-source software for running AI models more efficiently. It was announced at the PyTorch Conference. It is aimed at teams that run their own AI models on many computers at once, not at people who simply use a chatbot.

What it is

Running a model to answer a request is called inference. llm-d is a toolkit for doing that across a group of machines. It runs on Kubernetes, a widely used system for managing software across many computers. Google says more than 500 contributors from 25 organizations worked on it. It is a Cloud Native Computing Foundation Sandbox project.

Why agents are the problem

Google says AI traffic is changing. A year ago, most of it was chat: one question, one answer. Now more work comes from agents, which are AI programs that take many steps and use tools. They send bursts of requests that often share long stretches of the same text.

Google lists three problems with older setups:

• Different machines redo the same work on the same prompts.
• Expensive chip memory fills up with duplicate data.
• One busy agent loop can slow down people waiting on a live answer.

What it does

llm-d acts like a traffic director. It sends each step of a task to the machine that already holds that task's context. It also queues bursty background work so live requests stay quick. It can move stored context to cheaper memory when chip memory runs low, instead of throwing it away.

Version 1.0 adds a tool for less urgent jobs. These pull from queues such as Google Cloud Pub/Sub or Redis and run in spare capacity. Google also says its time-slicing feature puts idle chips to work during AI training. It claims this raises usage from about 40% to over 70%.

The numbers

On an agentic coding test using Google's Ironwood TPU chips, Google says llm-d cut the median wait for a first response by more than 85%. It also raised total throughput 2.4 times, compared with a plain Kubernetes setup, as of October 2026. These are Google's own results on one workload.

Who it affects

The software is built for companies serving their own models. Teams can keep their existing engine, model and cluster and adopt only the pieces they need. Most small businesses use hosted AI services and will not install this. Still, if it makes AI serving cheaper for the companies that provide those services, the savings could eventually show up in the tools you use. That is a possibility, not something Google promised.

What is not known

The material does not say what it costs to run llm-d beyond the hardware and cloud use it needs. It also does not say how results compare outside Google's test or on other chips.