An engineering note from building Ada, Optilogic’s agentic AI system for supply chain design. No code, no internals — the idea is the transferable part, and it’s the one I’ve since wanted in every system I’ve touched.
The constraint
Every request into the system had to be fully self-describing, and the server was not allowed to be the authority on anything. For reasons I won’t go into here — they were good ones, and they weren’t mine to negotiate — statelessness was a requirement, not a preference.
That is a lovely property to operate. It is a miserable property to be fast in.
Because an agentic turn is not cheap to set up. Before Ada can answer anything about a customer’s supply chain, it has to resolve which data models the user is pointed at and build working interfaces over them, which means going out to the platform API and doing real work against real, large models. In a stateful design you’d do that once per session and forget about it. Stateless means you do it per turn. The user pays for it on every single message, forever.
The obvious fix is to hold a session in memory and keep the expensive things alive in it. That’s exactly the fix we couldn’t have.
The observation
So I stopped looking at the request and started looking at the sequence.
A turn looks like one event from the server’s point of view, but it never is. Watch a real user:
- They select the data models they want to work on. ← the expensive signal
- They adjust a setting or two.
- They type. For a while. And eventually hit enter.
The costly input arrives first, and then the user spends five, ten, twenty seconds composing a question. From the server’s perspective those seconds are dead air. From the user’s perspective they’re not waiting at all — they’re working.
Everything expensive about the turn was knowable before the turn existed. We were just choosing to discover it late.
Temporary statefulness
So we split the two things that “stateless” usually gets conflated into: authority and memory.
The server stayed completely without authority. It never decides anything about configuration, never remembers what you told it last time, never fills in a blank from an earlier request. Every request still arrives complete, still replaces rather than merges, still fully describes the world it wants.
But it was allowed to remember work it already did — for a while, and never as a source of truth.
Selecting a data model stopped being a piece of information that the next turn would consume, and became an event that immediately kicks off building the expensive resources, in the background, while the user is still typing. By the time the message lands, the turn doesn’t build anything. It picks up work that finished seconds ago and gets straight to the part the user cares about.
That’s the whole trick: temporary statefulness with no authority. State as a cache of effort, never a cache of truth.
Two rules make it safe, and they’re the part I’d defend hardest:
Warm equals cold. A request against a warm process and the same request against a freshly started one must produce byte-identical behaviour. The warm process is faster. It is never different. This is what makes the optimization something you can reason about at 3am: if the cache is the only thing that differs, the cache can never be the bug.
Warming is best-effort and invalidation is explicit. If a warmup fails, is slow, or never happens, the turn is slower and completely correct — the warm path and the cold path run the same code, one just finds the work already done. And caches are dropped by deliberate, named decisions — the conversation was replaced, the selection changed — never inferred from object identity or clever comparisons. Identity-based invalidation is how you build something that’s correct on Tuesday and wrong on Wednesday.
Why I like this one
The latency the user feels went from every message pays for setup to setup is free, and we gave up nothing: no session affinity, no sticky routing, no server that has to be interrogated to understand why it answered the way it did. You can still restart the thing mid-conversation and nothing is lost, because nothing important was ever in there.
But the reason I tell this story in interviews isn’t the speed. It’s the shape of the problem.
The constraint was real and non-negotiable, and the first three ideas anyone has all involve removing it. The performance was sitting in the ordering of events — somewhere nobody looks, because everyone models a request as an atom. Most of the latency I’ve found in AI products since has been in the same place: not in the model, not in the code inside the turn, but in when the system chose to learn things it could have known earlier.
Stateless doesn’t have to mean amnesiac. It has to mean the server never gets a vote. Those are very different requirements, and the gap between them is where the free performance lives.