One of our chat apps lets you choose which model answers you, and for most of those choices it was fine. Point it at DeepSeek running on Azure, though, ask a reasoning-heavy question, and the screen would freeze for anywhere from a few seconds to most of a minute before the first word appeared. The strange part was that the same DeepSeek family, reached through a different provider and passed through the very same plumbing, felt instant. So the useful question was never “is DeepSeek slow.” It was “why does one path to the same model hang while another flies?” This post is the investigation that answered that, and the design we settled on to stop it happening again.
The short version, for the impatient: the delay was upstream of anything we built, it had two independent causes, and the fix is not a faster model but a small loop that quietly holds the right models warm while a human is actually using them. The longer version is more interesting, because chasing this down changed how we think about latency across a fleet of providers.
Some context: Lattice, and the thing in the middle
We build Lattice, an agentic analyst you talk to in plain language. You ask a question, it works out which of your data sources are relevant, writes and runs code, checks its own results, and hands back a chart or a written answer. The model doing that thinking is one you choose and can swap at will: a model on your own hardware, a lower-cost cloud model such as DeepSeek or Kimi, or a top-tier model such as Claude or GPT.
Serving that many kinds of model behind one plain chat interface takes a layer in the middle, and that layer is where this story happens. We call the whole thing the harness. A harness, in the sense we mean it, is the scaffolding that wraps a set of models and makes them usable as one: it takes each request from the client, routes it to whichever provider and model you picked, speaks that provider’s particular dialect, normalizes the streaming reply into one shape the client understands, and manages the connection back to you, token by token. Every request drives down that one road, whichever model it is aimed at, which will matter later, both for finding the problem and for fixing it.
When something in that setup feels slow, there is a natural temptation to blame the newest and most complicated piece you own. The harness is new, it is complicated, and it is ours, so it was the obvious suspect. That instinct turned out to be exactly wrong, and the way we proved it is worth writing down.
How we measured, and why the usual number is the wrong one
The reflexive way to measure latency is “time to first byte,” meaning how long after you send the request the first scrap of data comes back. That number is nearly useless here, because the first byte a user gets is often a keepalive ping, a small heartbeat the harness sends to say “still working,” not a word of the actual answer. A stream can start its first byte in fifty milliseconds and still leave the human staring at nothing for forty seconds.
So we measured the number that matches what a person actually feels: time to first content token, the wait until the first real piece of the answer appears. Alongside it we recorded three things that each rule a suspect in or out:
- Maximum gap between chunks, once content is flowing. If the harness were secretly buffering the response, holding tokens back and releasing them in a clump, we would see one long silence followed by a burst. A steady drip with small gaps means no buffering.
- How many keepalive pings went out before the first real token. This tells us how long the harness sat waiting on a silent upstream, which is the delay it is papering over rather than causing.
- Whether the model’s reasoning streamed at all. Modern reasoning models produce a visible chain of thought before the final answer. Whether those “thinking” characters stream live, or the model goes dark until the answer is ready, turns out to be half the story.
We ran two workloads. A multi-step reasoning prompt, capped at 1,500 tokens, to compare paths under realistic load, and a tiny arithmetic prompt, capped at 64 tokens, so that the cold-versus-warm number would reflect startup cost rather than how long the answer took to generate. Every probe went through the real request paths, once straight to the provider and once through our harness, so we could put the two side by side. API keys were never logged, only their length, and the prompts were benign.
The finding, in one table
Here is the heart of it. Each cell is the time to first content token over three rounds, shown as fastest, median, and slowest, next to the largest mid-stream gap and how much live reasoning the model streamed.
| Path | First content (min / med / max) | Max mid-stream gap | Reasoning streamed |
|---|---|---|---|
| DeepSeek on Azure, direct to provider | 10.1 / 10.6 / 41.1 s | 0.16 s | 0 characters |
| DeepSeek on Azure, through our harness | 7.8 / 11.0 / 14.0 s | 0.16 s | 0 characters (2 to 3 pings first) |
| Native deepseek.com, direct to provider | 0.7 / 0.8 / 1.4 s | 0.20 s | 3,100 to 3,600 characters |
| Native deepseek.com, through our harness | 0.8 / 0.9 / 0.9 s | 0.18 s | 2,200 to 3,000 characters (1 ping first) |
Three things fall straight out of it.
First, the harness is not adding the delay. The Azure path takes eight to forty-one seconds to first content whether you go through the harness or straight to the provider, and the single worst call, at forty-one seconds, was the direct one. If our plumbing were the culprit, the direct path would be the fast one. It was not.
Second, the harness is not buffering. The largest gap between chunks, once content starts, is about 0.16 seconds in every cell, direct or not. A buffered stream would betray itself as one late clump. A steady 0.16-second drip is the opposite of that. It is proof the harness forwards each token the instant it arrives.
Third, and this is the part that surprised us, the Azure deployment streams zero reasoning. Native DeepSeek pours out thousands of characters of live chain of thought, starting under a second in. The Azure deployment of the same model family emits none at all, then hands over the finished answer. Even when it is warm and fast to start, it is silent for the entire thinking phase.
Two kinds of silence
That last point deserves its own name, because it reframes the whole problem. A frozen screen can mean two completely different things, and to the user they look identical.
The first kind of silence is cold-start silence: nothing is happening yet. The model instance had been idle, was shut down to save money, and has to be spun back up before it can produce a single token. Nothing you do at the client end fills that gap, because there genuinely is no output to show.
The second kind is thinking silence: a great deal is happening, you just cannot see it. The model is reasoning hard, but the deployment does not stream those reasoning tokens, so the wire is quiet until the conclusion is ready. This silence is avoidable, and native DeepSeek avoids it, by streaming the chain of thought as it goes.
Our hanging app was suffering both at once. The Azure path could be cold, which is the first silence, and even warm it never streams its reasoning, which is the second. Native DeepSeek has neither problem: no cold start, and a chain of thought that fills the screen within about a second. That is the entire reason one path feels smooth and the other feels broken, and neither reason has anything to do with the model’s underlying quality. This lines up with something we learned building the analyst loop: a wait that is narrated feels like watching someone competent work, while the same wait behind a blank spinner feels like a crash. Streaming the reasoning is not a cosmetic nicety. It is the difference between the two kinds of silence.
Confirming the harness’s innocence, and what it already does right
Numbers pointed away from the harness. Reading its code confirmed it, and turned up that it was already doing the right thing under the hood.
It reads the upstream response line by line and forwards each translated event immediately, with no buffer in between, which is why we saw that steady 0.16-second drip rather than a clump. When a provider does send reasoning tokens, the harness maps them into the client’s “thinking” blocks, which is exactly why native DeepSeek shows its live chain of thought and the Azure deployment shows none: the harness faithfully relays what it is given, and Azure gives it nothing to relay.
The more interesting piece is how the harness handles a silent upstream, because it has to make a genuine tradeoff. For the first couple of seconds it holds off committing to a response, so that a fast upstream failure, say a rate-limit rejection, can still come back as a real error status the client can act on. After that grace window it commits to a streaming response and starts sending a keepalive ping immediately, then another every few seconds for as long as the upstream stays quiet. The reason is not decoration. Without those pings, newer client builds treat a long silence as a dead connection, give up, and re-send the request, which piles a duplicate onto an already-overloaded deployment and makes everything worse. The pings keep the connection honestly alive so the client waits instead of stampeding.
One detail mattered for what came next: this pinging is scoped to a single turn. It exists only while a request is in flight, and the moment the turn ends or the client disconnects, it stops and cancels the upstream work behind it. There is no background heartbeat idling in the harness. That is the correct default, but it is also precisely the thing we will need to extend, carefully, to keep models warm between turns.
The real unit is a model instance, not a model
Here is the idea that reorganized our thinking. We are used to talking as if speed were a property of a model. “DeepSeek is fast.” “Kimi is stable.” That habit is what kept the investigation confused for a while, and it is simply wrong. Latency is a property of a specific model instance and where it is hosted, not of the model’s name and not even of the provider it sits behind.
Two facts from our own setup make this concrete, and both were quietly causing confusion before we saw them clearly.
The first: Kimi is dual-hosted. We reach one Kimi through a serverless Azure deployment, which is cold-prone, and another through a managed provider that keeps it always warm. So “Kimi is stable” was true and misleading at the same time. The stable one was the managed instance. The model was never the point.
The second, and the sharper one: a single Azure account serves both a warm model and a cold one. The same provider hosts pooled GPT deployments, which Azure keeps warm and which never scale to zero, right alongside serverless DeepSeek deployments, which do scale to zero and pay the cold-start tax. One provider, one account, two completely different latency personalities depending on which model you ask for. You cannot reason about this “by provider” any more than you can “by model.”
Underneath both facts is a bargain worth naming. Serverless “scale to zero” hosting means that when a model sits idle it is shut down entirely, so you pay nothing for it. In exchange, the next request after an idle stretch has to wait for it to spin back up, and that is the eight-to-forty-one-second cold start we measured. For batch work that bargain is excellent. For an interactive chat with a human watching a blank screen, it is a bad trade, because the tax lands exactly when someone is waiting.
Sorting our providers by that bargain rather than by name gives a clean picture:
| Hosting class | Examples | Cold-prone? | Why |
|---|---|---|---|
| Managed inference | Native deepseek.com, NVIDIA, OpenAI, Anthropic, Gemini, and Azure’s pooled GPT deployments | No | the provider keeps capacity warm and pooled for you |
| Serverless, scale-to-zero | DeepSeek, Kimi, and Claude when hosted on Azure’s serverless tier | Yes: rare but huge | idle instances are shut down; the first request after idle waits 8 to 41 s |
| Self-hosted | a 35B model on our own Ollama box | Yes: frequent but modest | about 2.5 to 3 s whenever its idle timer lapses or another model evicts it from memory |
The two cold-prone rows fail in opposite rhythms, which shapes the fix. The serverless spike is rare and enormous, near-instant most of the time and then a huge stall after eviction, so it is hard to predict but very valuable to prevent. The self-hosted stall is small but constant, a few seconds every time the model’s short idle timer lapses, so it is low value per hit but it recurs endlessly on a quiet app. We could not force the serverless cold start on demand, because when the instance scales back down is Azure’s decision, not ours. But the self-hosted box can be forced cold by unloading the model, and it reports its own load time honestly: about 2.58 seconds cold against 0.05 seconds warm, and that was a warm-disk reload, so a truly cold load would be larger still.
A field note from that self-hosted box, in case it saves someone an afternoon: it sits behind an Open WebUI proxy rather than raw Ollama, so the standard endpoints are not where you expect. The OpenAI-compatible route lives at a nonstandard path, and the native Ollama passthrough is the one that lets you force an unload and then read the true load time back out of the final frame. The proxy also uses a self-signed certificate, so the client has to be told to skip verification for that host. None of this is exotic, but every piece of it was a small surprise in the moment.
Keep-warm: holding the door open, but only when someone is coming
If cold start is the cost of a model having been idle, the fix is to not let the right models go idle while a human is around. That is all keep-warm is: a small loop that sends a cheap, one-token request to a cold-prone model often enough to reset its idle timer, so the human’s next real question lands on an already-warm instance. The whole design is in how narrowly you scope that, because holding a model warm spends real compute, and spending it carelessly is its own kind of waste.
The keep-warm loop has four guardrails.
Off by default. With nothing configured, the loop does not exist. No pings, no behavior change, no surprise compute bill. It only wakes up when someone deliberately turns it on.
Activity-triggered, never speculative. This is the ethical and economic core of the thing. We only keep a model warm if a real user actually used it recently, within an active window of, say, half an hour. No traffic, no pinging. We are not holding every model warm on the off chance. We are holding the door open for the person who just walked through it and is likely to come back for a follow-up. The moment they stop, the pinging winds down on its own.
Per instance, not per model or per provider. This is the direct consequence of the earlier insight. Because latency lives at the instance, so must the warming. The loop targets the specific cold-prone instances, the serverless DeepSeek, Kimi, and Claude deployments and the self-hosted models, and it deliberately skips the warm ones even when they share an account with a cold one. It never pings a managed provider, which gains nothing, and never a warm pooled GPT deployment that happens to sit behind the same Azure account as a cold DeepSeek.
Respect the capacity you are pinning. On a memory-bound self-hosted box, holding one model in memory can evict another, so the loop warms only the model the app actually uses, not every model on the machine. Warming the wrong thing there does not just waste effort, it actively pushes out the thing you wanted warm.
The mechanism follows the shape of the harness. Because every request already flows through that one road, we record a timestamp there each time a model is used: this instance was wanted, just now. A background loop then wakes on an interval shorter than the shortest eviction timeout and, for each cold-prone instance that was used inside the active window and has not been pinged recently enough, sends one minimal warm request. The lever differs by host. A self-hosted Ollama model takes a one-token request with its keep-alive set long, which both loads it and resets the timer. A serverless deployment has no keep-alive knob at all, so the warm request itself, a single-token completion through the harness’s normal request path, is the only lever, and it works because it resets the idle clock the same way a real request would.
A few guardrails come baked in. The loop selects its own targets, picking out the cold-prone instances with recent activity so nobody has to list them by hand, and it paces its pings comfortably under the eviction timeouts we measured. Every warm ping is capped at a single token and tagged in the traffic log so it can never be mistaken for user activity, the loop honors each provider’s rate limits and certificate settings, and managed or administratively disabled providers are never touched. It is off unless someone deliberately switches it on, and turned off, none of the normal paths through the app notice it exists at all.
What keep-warm does not replace
It would be dishonest to sell this as the whole answer, so here is where it stops. Keep-warm is a cheap approximation of a warm instance, not a guarantee of one. If a use case genuinely needs a latency SLA, the structural fix is a provisioned deployment that never scales to zero in the first place, and keep-warm is the budget version of that. Where data residency does not force the choice, the simplest fix of all is to route latency-sensitive reasoning to a managed provider like native deepseek.com and keep the serverless Azure deployment for work that has to stay in that environment. And for the occasional cold spike that slips through anyway, a single automatic retry on the first token often lands on a now-warm instance, because the first doomed request did the work of waking it. These are complementary. The right production posture is usually a couple of them at once, with keep-warm as the always-on floor.
The bottom line
When you serve a fleet of providers behind one interface, you inherit an uneven latency surface, and no amount of picking a “faster model” flattens it, because the unevenness is not in the models. It is in where and how each one is hosted. Making that fleet feel uniform to a person typing a question is an infrastructure problem, and it comes down to three habits that all say the same thing: never leave someone staring at an unexplained frozen screen.
Send a keepalive so a slow upstream reads as patience rather than a dead line. Stream the model’s reasoning so that thinking silence becomes visible work instead of a stall. And keep the cold-prone instances warm while a human is actually there, so the expensive first-token tax is quietly paid before anyone has to wait for it. The harness we suspected turned out to be doing the first of those already, and doing it correctly. The other two are the work this investigation pointed us toward.
The durable lesson is the one that took us the longest to say plainly. “Fast” is not a property you can read off a model’s spec sheet. It is a property of a deployment, and smoothness is something you engineer around the model rather than something you select instead of it.
Reproduce it yourself
If you want to check any of this against your own setup, the method is simple. Run a streaming client against each target and record the time to first content token, not first byte, along with the largest gap between chunks and, on the harness path, how many keepalive pings arrived before real content. Force a self-hosted model cold by unloading it, then read its reported load time out of the final streamed frame; that number is authoritative in a way a wall-clock stopwatch is not. A serverless cold start cannot be forced, since the provider decides when to scale down, so catch it opportunistically after the deployment has sat idle. Keep the prompts benign, log key lengths rather than keys, and honor each provider’s certificate settings. The gap between the two silences shows up immediately, and once you can see it, the rest of the design more or less writes itself.