What a Cell Knows About Throughput

A yeast cell swimming in sugar, with oxygen all around it, will do something that looks insane on a spreadsheet.

It ferments.

Fermentation gets you 2 ATP per glucose molecule. Respiration gets you around 30. The cell has the oxygen. It has the mitochondria, fully built, sitting right there. And it picks the option that is fifteen times worse.

This isn't a bug in yeast. E. coli does it too — feed it well and it starts dumping acetate, which is basically throwing away half-burned fuel. Fast-growing tumor cells do it, which is where the name comes from: the Warburg effect. Otto Warburg saw it in the 1920s and assumed the mitochondria must be broken. They aren't.

The mistake is in the denominator. ATP per glucose is the wrong metric when glucose isn't the thing you're short of. The metric the cell is actually optimizing is closer to ATP per second per gram of protein you had to build and carry around. Respiration is a large, slow, expensive apparatus — dozens of proteins, membrane real estate, iron-sulfur clusters, the works. Fermentation is a short pathway of cheap enzymes running very fast. When your binding constraint is how much machinery you can afford, the wasteful path wins on the metric that counts.

I think about this every time someone in a design review says "but that's redundant compute."

The premise

The job, stripped down, is this: serve as much traffic as you can from a fixed budget of machines, and make sure the traffic you serve is worth serving. Every architecture decision I've made in the last few years reduces to some version of that. Where do I put the cache. Which service do I scale. Do we shed this traffic or eat the cost. Is this migration worth six weeks.

Cells have been solving that exact problem for about three and a half billion years, under harsher constraints than mine, and biochemists have spent seventy years writing down how. Some of what they found is obvious once translated. Some of it isn't, and those are the parts worth your time.

The mapping I'll use throughout:

One correction to the obvious version of this, because it changes the economics. Enzymes are catalysts. They are not consumed, and catalysis itself is close to free. What's expensive is building the enzyme — protein synthesis is one of the largest line items in a cell's budget, and the cell pays it whether or not any substrate ever shows up. Which is the right model for your services too. The EC2 bill doesn't care how many requests you got.

1. The cheap path and the fast path are different paths

Back to fermentation. What the cell is doing is spending substrate to save proteome. It has more glucose than it has room for machinery, so it picks the pathway with the worse yield and the better rate.

You do this constantly, and you probably feel vaguely guilty about it.

Caching is fermentation. You store the same answer in four places, none of which is the source of truth, and you pay for that in memory and staleness and invalidation bugs. Denormalization is fermentation. You duplicate the customer's country code into eleven tables because the join was costing you 40ms. Read replicas, materialized views, precomputed feeds, fanout-on-write — all of it is buying latency with resources you happen to have in surplus.

The useful reframe isn't "waste is fine." It's: name your surplus before you optimize. Cells don't ferment when glucose is scarce; they switch straight back to respiration. The pathway that's correct at one resource ratio is wrong at another. If storage is cheap and your p99 is what's killing you, spend storage. If you're memory-bound on a fleet you can't grow, the elegant low-footprint path is worth its complexity. Same system, different answer, six months apart.

Twitter's timeline is the cleanest public example I know of. The numbers are from Raffi Krikorian's 2013 architecture talk, and Twitter is a different company now, but the decision has the right shape: roughly 6,000 tweet writes per second against 300,000 timeline reads per second. Fifty to one.

The tidy design computes the timeline at read time — look up everyone this user follows, fetch their recent tweets, merge, sort. It is also the design that does the expensive work three hundred thousand times a second.

So they don't. Every tweet is fanned out at write time into a precomputed home timeline for each follower, held in a Redis cluster, capped at 800 entries, replicated three times across three machines. An account with 20,000 followers generates as many as 20,000 inserts for a single tweet. Delivering one tweet to a million followers takes 3.5 seconds at p50 and up to five minutes at p99.

What that buys is a read that is a lookup rather than a computation: 5ms median to the timeline service, 100ms at p99.

And the surplus runs out exactly where you would predict. Fanning out to a Taylor Swift-sized account costs more than the reads it saves, so those accounts stopped being fanned out at all and get merged in at read time instead. Same system, both pathways running side by side, and the boundary between them sits precisely where the follower count makes the trade flip.

2. There is usually no bottleneck

This is the one that changed how I read dashboards.

In the 1970s, Henrik Kacser and Jim Burns worked out something called Metabolic Control Analysis, and it starts from a question that sounds trivial: if I speed up one enzyme in a pathway by 1%, how much does total pathway throughput go up? The answer is that enzyme's flux control coefficient. And there's a theorem: across the whole pathway, the coefficients sum to 1.

Sit with that. It means control is a fixed budget that gets divided. If one enzyme has a coefficient of 0.9 — a genuine bottleneck, where speeding it up nearly translates one-to-one into throughput — then everything else in the pathway shares the remaining 0.1. But that's the unusual case. Measured in real pathways, control is typically smeared: 0.3 here, 0.25 there, 0.2, 0.15. Nobody is the bottleneck. Doubling any single enzyme buys you almost nothing.

The folk model most of us carry around — find the bottleneck, fix the bottleneck, find the next one — describes a system in a specific and fairly rare state. Systems that have already been tuned a few times don't look like that. They look like four services each holding a quarter of the control, where the only way to get 30% more throughput is to touch all four.

The operational version: before you scale a service, get an estimate of its control coefficient. Perturb it a little in production and measure end-to-end throughput, not the service's own latency. If a 20% capacity increase on that service moves total throughput by 3%, its coefficient is around 0.15, and the sprint you were about to spend optimizing it will produce a result nobody can see on a graph.

3. The only lever with a coefficient of 1

I left a corollary out of the last section.

Speeding up one enzyme buys you its control coefficient, and if that's 0.25 then that's what you get. But there is exactly one perturbation that returns a coefficient of 1, and the summation theorem is what guarantees it: increase every enzyme in the pathway by the same factor, and flux increases by that factor. Exactly. No diminishing return. No bottleneck relocating somewhere annoying.

Copy the whole pathway.

Which is what cell division is. A bacterium in rich medium does not sit there tuning its hexokinase. It builds a second complete copy of itself and splits. You will never find a bacterium that responded to abundance by doubling one enzyme, because the only move that reliably doubles throughput is doubling all of it at once.

That's your horizontally scaled fleet, and it's the mathematical reason replicating whole units — pods, containers, service groups, whatever your deployment atom is — keeps outperforming per-service tuning. Everything else divides a fixed budget of control. This is the one thing that doesn't.

The lag is the machinery

Drop bacteria into fresh medium and nothing happens for a while. That's lag phase, and the cells aren't idle during it — they're building ribosomes, transporters, and whatever enzymes the new carbon source demands. The machinery for growth has to exist before growth can happen. Only then does the exponential phase start.

Cold start. And the biology has something specific to say about it: lag duration depends on how different the new medium is from the old one. Move a cell between near-identical media and it barely pauses. Move it onto a novel carbon source and it can sit there for the better part of an hour re-tooling. The closer your base image is to what production actually needs at runtime, the shorter the lag. Every dependency you fetch on boot is re-tooling you chose to do in the critical path.

The harder constraint: E. coli's doubling time bottoms out around twenty minutes even under ideal conditions. That's a floor, and no amount of nutrient abundance beats it. Which means any demand spike arriving faster than your scaling rate cannot be answered by scaling — it has to be absorbed by a buffer or shed at the door. That's the reason autoscaling has never once saved anybody from a thundering herd. Growth rate has a ceiling. The spike doesn't care.

Overshoot

Populations that grow past what the resources support don't glide down to equilibrium. They crash, frequently to below what steadier growth would have sustained. The famous case is at a different scale entirely, but it's hard to beat: 29 reindeer introduced to St. Matthew Island in 1944 became roughly 6,000 by 1963, then about 42 by 1966. They ate the lichen faster than it could regrow, and the lichen took decades.

Autoscaler thrash is this exact shape. The spike triggers scale-out, the new instances pile onto a shared dependency that didn't scale with them — connection pool, primary database, some downstream service on a fixed capacity — and now everything is slower, which reads as load, which triggers more scaling. A growth signal that can't see the shared constraint will always overshoot it. The fix, in reindeer and in fleets, is to make the constraint legible to whatever is making the growth decision.

Slack's outage on January 4th, 2021 is that shape drawn from life, and it has a detail I would not have predicted.

Traffic that morning was unusually heavy — first Monday back after the holidays, client caches cold, everyone pulling down more data than normal. An AWS Transit Gateway between their VPCs saturated and began dropping packets. Web-tier threads then spent longer waiting on slow backend calls, and waiting threads don't burn CPU. Utilization fell. The autoscaler read low CPU as over-provisioning and scaled the web tier down, in the middle of a load spike.

Thread utilization on the now-smaller fleet spiked, and the autoscaler reversed hard: 1,200 servers requested between 7:01am and 7:15am PST. Every one of them went through provision-service, which configures and tests new instances — and provision-service reached its own dependencies over the same degraded network, hit the Linux open file limit, and fell over. The instances came up unprovisioned and never served a request. They still counted against the autoscaling group ceiling, so Slack hit its size limit with a fleet made largely of servers doing nothing. AWS raised the gateway capacity manually at 10:40am.

Three failures of legibility in one incident. The growth signal measured the wrong quantity, and pointed backwards. The growth action loaded a shared dependency nobody had sized for a 1,200-instance burst. And that dependency was sitting on the same degraded resource that triggered the whole thing.

4. Scaling down runs a different program

The tidy version of this analogy says that when resources get scarce, cells die off until the survivors match what's left. It's a clean story and it's mostly wrong, and what actually happens is more useful to you than what people assume happens.

When E. coli runs out of glucose, it doesn't begin dying. It changes sigma factor. RpoS takes over a large share of transcription and runs a different program entirely: the cell shrinks, condenses its chromosome, switches on stress responses, and becomes markedly more resistant to heat, acid, oxidative damage and osmotic shock than it was ten minutes earlier while growing. Stationary phase is not a growing cell with the volume turned down. It's a different machine, built on purpose, out of the same parts.

That's the part worth stealing outright. Degraded mode should be a program you designed, not a state you end up in. Most systems treat reduced capacity as "the normal service, but there's less of it," and then find out under load that the normal service quietly assumed things that only hold at full size — a warm cache, an available replica, a batch job that's been keeping a table pruned. The cell's answer is to stop growth entirely and switch on machinery it wasn't running before. The equivalent is a written configuration: which endpoints stay up, what you stop precomputing, which timeouts tighten, what starts getting rejected at the edge. Deployed and exercised, ideally on a schedule. Not reconstructed at 3am from whatever happens to still be responding.

Three ways to rest

Bacillus subtilis takes the extreme option and sporulates. Sporulation is slow to commit to, costly, and largely irreversible once the decision is made — but it yields something with essentially zero metabolic cost that survives boiling, desiccation and radiation. Waking up is correspondingly slow.

So there are three resting states, and they sit along a curve:

• Growing — full cost, no resume latency
• Stationary — reduced cost, fast resume
• Spore — near-zero cost, slow and expensive resume

Which is the same curve you're picking from with a warm fleet, a scaled-down minimum, and cold serverless. The cell chooses by betting on how long the famine runs. So are you, whether or not you've framed the decision that way. The question was never which option is cheapest. It's how long the quiet lasts, and what it costs you to be slow at the moment it ends.

Worth noting that the cell doesn't sporulate at the first sign of trouble. Commitment is gated behind sustained starvation signals, specifically because the resume cost is so high. Your scale-to-zero threshold wants the same asymmetry: quick to add capacity, slow and deliberate about dropping to nothing.

The maintenance floor

Cells spend energy simply staying alive, independent of any growth — holding membrane potential, repairing proteins, maintaining gradients. Below a threshold substrate concentration they can't cover even that, and that is the point at which death actually enters the story.

Your fleet has this number too: what it costs to exist at zero traffic. Most teams have never measured it, which means they've never noticed the moment it stopped being a rounding error against their variable cost. If most of your bill is invariant to traffic, a good chunk of the optimization work on your roadmap is aimed at the wrong term.

Survival isn't a merit question

The last piece of the intuition to take apart is the idea that survivors are the ones best suited to the remaining resources.

Bacterial survival under starvation and under antibiotic exposure is substantially stochastic. Persister cells are a small subpopulation that drops into dormancy before any crisis arrives — not because they sensed something, and not because they're fitter, but as a hedge. They pay for it during good times, because a dormant cell isn't growing. What the population buys is survival of events that kill everything metabolically active. And persisters survive those events precisely because they weren't doing the thing that gets attacked.

Two consequences.

Random survival is exactly what an unmanaged scale-in hands you, which is why scale-in deserves design attention that scale-out usually monopolizes. Which instances terminate is a decision that's available to you: connection draining, oldest-first, zone balance, don't kill the one holding the lease or the one whose cache took an hour to fill. Leave it to chance and you'll lose the wrong ones at the worst possible time.

The second consequence is the real argument for cell-based architecture. Persisters survive by being isolated from a failure mode through not participating in it. When a poison-pill request or a bad deploy takes out everything currently serving, what saves you is capacity that wasn't serving. Holding a fraction of your fleet deliberately outside the blast radius — a later deployment wave, a separate shard, a group not yet taking the new traffic pattern — costs you during the good times in exactly the way a persister does. That cost isn't waste. It's the premium.

Two well-documented cases of that premium going unpaid.

Cloudflare, July 2nd 2019. A WAF rule containing a regular expression that backtracked catastrophically was deployed globally in one go. CPU hit 100% on machines worldwide, traffic across the network dropped 82% at peak, and the global 502s ran from 13:42 UTC until they killed the WAF at 14:09. Twenty-seven minutes. Nothing had been held back, so there was no unaffected fraction to shift traffic onto — recovery meant switching the feature off everywhere.

CrowdStrike, July 19th 2024, is the same failure at a scale that makes the point hard to argue with. A Rapid Response Content update, Channel File 291, supplied 21 input fields to a sensor that expected 20. Out-of-bounds read, kernel crash, and by Microsoft's count around 8.5 million Windows machines down.

The detail worth keeping is how the affected population is defined in CrowdStrike's own root cause analysis: hosts that were online and downloaded the file between 04:09 and 05:27 UTC. Seventy-eight minutes. Everything that came through it came through by not participating — powered off, on a network that could not reach the update, otherwise not doing the thing that was under attack.

That is a persister population, and it is an accident. The argument for cell-based architecture is that you should be able to produce one on purpose.

5. Backpressure has a name and it's older than you

Pathways regulate themselves, and not by dropping work at the far end. The mechanism is feedback inhibition: the final product of a pathway binds to an enzyme near the beginning and shuts it down.

The textbook case is isoleucine. Its own pathway's first committed step is catalyzed by threonine deaminase, and isoleucine binds to that enzyme and inhibits it. When there's plenty of isoleucine around, the cell stops committing threonine to making more. Not at step five. At step one, before the raw material has been spent.

Notice where the signal travels: backwards, against the flow, from the point of accumulation to the point of admission. That is exactly the property that makes backpressure work and exactly the property most systems are missing. Queues fill up at the slow service and the fast service upstream keeps cheerfully accepting work, converting available memory into a longer queue and a worse p99, right up until something falls over.

The half-measure most teams ship is a retry budget and a circuit breaker at the caller. That's better than nothing, but it's local — it protects the caller from the callee. Feedback inhibition is global: the pathway's saturation is what throttles the entry point. If your queue depth at step four doesn't influence your admission decision at step one, you don't have backpressure. You have a buffer and some optimism.

6. Evolution doesn't refactor in place

The version of this that people reach for is mutation: a better enzyme appears, the old one gets replaced, natural selection sorts it out. That's not really how new enzymes arise, and the real mechanism is a much better fit for what you actually do.

It's duplication and divergence. The gene gets copied. Now there are two identical copies doing the same job, which is redundant and slightly wasteful. But because copy A is still handling the load, copy B is free to accumulate changes that would have been fatal in a single-copy world. Most of the time B degrades into junk. Sometimes it lands on something new, and now the organism has two enzymes where it had one.

This is blue/green. It's also the strangler fig, and dual writes, and shadow traffic. All of them are the same trick: pay for redundancy to buy yourself a copy that's allowed to fail.

And notice what the biology says about the failure rate. Most duplicates die. The strategy is only rational because the copy is cheap and the downside is bounded — the original never stopped serving. If your migration plan has no cheap way to abandon the new path, you haven't built a duplicate. You've built a mutation, and you're betting the organism.

7. Spend at the door

Not all traffic is worth serving, and the cell's answer to this is aggressive and early.

Membrane transporters are selective — most molecules simply don't get in. And even for the ones that do, E. coli running the lac operon won't build the lactose-digesting machinery at all while glucose is available. Glucose is the better substrate; catabolite repression keeps the cell from spending protein on the inferior one until the good stuff runs out. That's not rate limiting. That's a value judgment about traffic, made before any resources are committed.

The general rule: rejection cost should be minimized, and rejection should happen before commitment. A request you're going to drop at the fifth service has already cost you four services' worth of work, plus the queue slots it occupied. Auth at the edge, quota checks before fanout, cheap validity filters ahead of expensive enrichment. Scrapers and abusive clients are the obvious case, but the more interesting one is the low-value legitimate traffic — the free-tier batch job, the analytics backfill — which the cell handles by simply not building the pathway while something better is competing for the same machinery.

Where this breaks

Analogies are load-bearing right up until they aren't, so here's where I stop trusting this one.

Growth is the cell's objective. Replicas are your bill. That's a real asymmetry and it survives everything in Section 3: a bacterium in rich medium grows until something external stops it, because more copies is the win condition. No autoscaler has ever wanted more instances. Your fleet growing is a thing you tolerate to serve demand, and the cell has no equivalent of the question "could we do this with fewer?" — which happens to be the most valuable question anyone asks in a capacity review.

The related divergence: a cell will readily destroy itself on behalf of copies of its genes elsewhere. Apoptosis, altruistic suicide during infection, the whole germline logic. Your company will not, and shouldn't.

Cells can't refactor. Evolution has no ability to tear down a pathway and rebuild it cleanly, so metabolism is full of baroque historical accidents that persist because every intermediate step had to be viable. You can take downtime. You can rewrite the thing. Don't romanticize a design process whose defining constraint is that it can never start over.

And there's no customer. No SLA, no contract, no one who is specifically angry that their request was the one that got shed. Load shedding in a cell is a statistical event. In a system it's a person, and the fairness questions that creates have no biological analogue at all.

Use the parts that transfer. Feedback inhibition, distributed control, replicate-the-whole-pathway, degraded mode as a designed program, duplication before divergence, admission before commitment. Those are structural facts about pathways under resource constraints, and your architecture is a pathway under resource constraints.

The rest is decoration.