Apple engineers explain how to run AI agents at scale for real users

Two Apple engineers have laid out what it takes to move an AI agent from a working demo to a system that serves real users. They spoke at QCon AI, and InfoQ published the session. Their message: the demo is the easy part. Safety, traffic and quality checks are the real work.

Who spoke and what they used

The speakers are Deepak Mohanakumar Chandramouli, a senior ML engineering leader at Apple, and Bhumik Vinodkumar Thakkar, a senior software engineer at Apple. They built their example on Ray, an open-source tool that spreads heavy computing work across many machines. It is mainly used for training AI models, fine-tuning them and running them. The talk shows it can also run agents, meaning AI programs that plan steps and use tools.

The trip planner example

The demo is a trip planner. A user asks for a three-day winter trip. The AI model does not just answer. It makes a plan, then calls a search tool that finds destinations by season and a weather tool that checks conditions in a city. A third piece pulls it all into a day-by-day itinerary.

The speakers also noted two ways to steer an agent. A fixed flow is predictable. Letting the AI model choose its own steps is more flexible but less predictable. Each has trade-offs.

What production adds

According to the session summary, the move from a local prototype to a production system on Kubernetes, a common system for running software across many servers, brings several concerns:

• Safety checks at each stage: the user request, planning, tool use and the final summary
• Observability, meaning inspecting traces to see how the agent reasoned
• Handling memory, so the agent can personalize answers
• Automatic scaling when traffic rises and falls
• Running in more than one region
• Using one AI model to grade another's answers, called LLM-as-a-judge

What is not known yet

The available text covers the setup and the building blocks, but cuts off before the detailed lessons. It gives no performance numbers, costs or results from Apple's own products. Readers who want the specifics on cluster design, memory and grading will need the full presentation on InfoQ.