Building Zero-Latency Agents & Neural Workflows

An agent that answers in four seconds feels broken even when it is correct. Users abandon, retry, and double-submit. Below roughly a second, the same agent feels like a feature. The gap between those two experiences is almost entirely engineering, not model choice.

This is a breakdown of where the time actually goes in a tool-calling loop, and the four changes that moved our median turn from 4.2 s to 1.1 s on the same underlying model.

Where the time goes

A single agent turn with one tool call is not one model request. It is at minimum two, with I/O in between:

user message
  ├─ 1. reasoning pass      → decides which tool to call     ~700ms
  ├─ 2. tool execution      → your database, API, or search  ~150ms
  └─ 3. synthesis pass      → turns tool output into prose  ~1400ms
                                                     total ≈ 2250ms

Two observations follow immediately. First, your own infrastructure is usually the smallest slice — optimising the database query is rarely where the win is. Second, the synthesis pass dominates, because it generates the most tokens, and generation is sequential.

Multi-step agents multiply this. Three tool calls in sequence is six model round trips. This is why agents that "think longer" feel exponentially slower rather than linearly slower.

1. Stream everything, including the wait

Streaming does not make the turn faster. It makes it feel roughly three times faster, which is the metric that matters. The trick is that most teams stream only the final synthesis and leave the user staring at a spinner through the first two phases.

Emit progress events from the whole pipeline, not just the last leg:

async function* runTurn(messages, tools) {
  yield { type: 'status', text: 'Understanding the question' };

  const plan = await model.stream({ messages, tools });
  for await (const chunk of plan) {
    if (chunk.type === 'tool_call') {
      yield { type: 'status', text: `Looking up ${chunk.name}` };
    }
  }

  const result = await executeTools(plan.toolCalls);
  yield { type: 'status', text: 'Composing the answer' };

  // Only this last phase produces visible prose
  for await (const token of model.stream({ messages, toolResults: result })) {
    yield { type: 'token', text: token };
  }
}

The user sees motion at ~200 ms instead of ~2200 ms. Nothing about the underlying work changed.

2. Parallelise tools, and let the model do it

The most common latency bug we find is a sequential await inside a loop over tool calls:

// Serial: 3 tools × 150ms = 450ms
for (const call of toolCalls) {
  results.push(await execute(call));
}

// Parallel: max(150ms) = 150ms
const results = await Promise.all(toolCalls.map(execute));

Modern models will emit several independent tool calls in one response if the tool descriptions make independence obvious. Two things encourage it: describe each tool as self-contained, and avoid phrasing that implies ordering ("after fetching the customer, get their orders"). Where a genuine dependency exists, keep it — but do not let a false dependency cost you a round trip.

For genuinely sequential chains, consider speculative execution: if one tool is called in 90% of turns, start it before the model asks and discard the result when it does not.

3. Route by difficulty

Not every turn needs the largest model. In the workloads we run, somewhere between half and three quarters of turns are classification, lookup, or a short factual answer over retrieved context — all of which a small fast model handles at equal quality and a fraction of the latency.

const SIMPLE = /^(what|when|who|how many|list|show|status)\b/i;

function pickModel(turn) {
  if (turn.toolResults && turn.tokensIn < 2000) return 'small';   // synthesis
  if (SIMPLE.test(turn.text) && !turn.needsPlanning) return 'small';
  return 'large';
}

A regex is a deliberately crude router and we would not ship it as the final design — but it is worth starting there, because it costs nothing and immediately tells you what proportion of traffic is actually easy. Replace it with a small classifier once you have the distribution.

The subtler win: the synthesis pass, which we established is the slowest phase, is almost always the easy one. The hard reasoning already happened. Routing synthesis to a small model is close to free.

4. Cache at three layers

Three distinct caches, often confused with one another:

Prompt caching

Your system prompt, tool definitions, and any large static context are re-sent on every request and re-processed every time. Provider-side prompt caching removes most of that cost. It requires keeping the cached prefix byte-identical and at the front — so no timestamps, no per-request IDs, and no reordered tool lists.

Tool result caching

Tool calls are frequently repeated within a session and across users. Cache on a normalised argument key with a short TTL:

const key = `${tool.name}:${stableStringify(tool.args)}:${tenantId}`;
const hit = await cache.get(key);
if (hit) return hit;                       // typically 40-60% hit rate
const value = await execute(tool);
await cache.set(key, value, { ttl: 60 });

Note tenantId in the key. Omitting it is how one customer ends up served another customer's data from cache — a bug that is both severe and very quiet.

Semantic caching

Near-duplicate questions ("what's our stock on hand?" and "how much stock do we have?") can share an answer. Embed the question, look for a neighbour above a similarity threshold, and reuse. Use this carefully: it is excellent for FAQ-shaped traffic and actively dangerous for anything time-sensitive or personalised. We gate it to read-only, non-personalised queries and keep the TTL short.

Measuring it properly

Percentiles, not averages. An average hides the turns that lose you users. Track at minimum:

• TTFT — time to first visible token, p50 and p95
• Turn duration — end to end, p50 and p95
• Tool count per turn — the strongest predictor of a slow turn
• Cache hit rate per layer
• Model mix — what fraction actually routed to small

Attach a trace ID at the edge and propagate it through every model call and tool execution. Without a trace you will optimise the phase that is easiest to see rather than the one that is slowest.

What it added up to

The single largest improvement in how the product feels came from streaming, which made the agent no faster at all. The largest real improvement came from routing, which is mostly a matter of admitting that most questions are not hard.

These numbers are from one workload — a document-heavy operations assistant with a small tool surface. Your distribution will differ. The order of attack usually does not: perception first, concurrency second, model size third, caching last.