I ran a spike to answer one question: can a federated GraphQL subgraph serve a real-time subscription through the production federation router to a browser, with authorization enforced, without falling over. The answer was yes. The things worth writing down are the three places where what I measured disagreed with what I had assumed.

the assumption

The capacity work in the plan assumed 5,000 concurrent open chats. Every downstream number came from that: memory per subscription times 5,000, and a “lazy resource allocation” workstream to keep the fleet-wide cost down.

The first version of the spike measured 234 KB of subgraph memory per held subscription, eight times the stand-in service I had tested against first. The reason was real: the request context factory builds all eighteen request-scoped data loaders on every request, and a subscription holds that context for the life of the stream rather than for one request. At 5,000 concurrent that is about 1.2 GB across the fleet. So the plan gained a phase-one item: build the loaders lazily, since the subscribe path touches two of eighteen.

Then I read production. The busiest hour in thirty days had 23 distinct active conversations. With about two members each, the highest concurrent-subscription count ever recorded would have been roughly 50. The assumption was a hundred times larger than anything that had happened.

Load tested at 50, 250, and 1,000 concurrent subscriptions, each with a distinct identity so the router could not collapse them: zero errors, latency flat at p50 39 ms and p95 44 ms even at 1,000, and 28 MB of subgraph memory added at twenty times the recorded peak, against pods with about 2.6 GB of headroom each. The marginal cost per subscription settled at about 28 KB once the heap had grown. The 234 KB figure included first-touch growth and overstated the steady state by eight times.

The lazy-loader workstream went from “phase one” to “optional, revisit if concurrent chats reach four figures”. A day of work, withdrawn by an afternoon of reading the tables the feature would actually serve.

the multiplexing model was wrong, and the conclusion survived anyway

The plan also assumed the router multiplexes every client subscription into roughly one upstream connection per router replica, so the subgraph would see 8 to 24 connections regardless of client count. Measured: the router’s dedup key includes the forwarded identity and the operation variables. Six clients on one token collapsed to two upstream connections. Two hundred clients with two hundred distinct identities produced exactly two hundred. The subgraph sees one upstream connection per concurrent (user, conversation) pair.

The conclusion that autoscaling needed no change survived, resting now on the measured per-connection cost rather than on a connection count that was wrong. That is the honest shape of a spike: the number you were most confident in is the one that turns out to be a guess, and the recommendation only holds if it holds for the measured reason too.

authorization at subscribe is not authorization while subscribed

Authorization runs once, when the client subscribes. I tested what that means directly. Member A subscribes. A’s membership is deleted while the stream stays open. Member B writes a message. A’s open stream receives the signal.

A fresh subscribe by the revoked member is correctly refused. But removal takes effect on the next subscribe, not on the open one, and in a quiet period the open one lives indefinitely, because the heartbeat keeps it alive.

What bounds the damage is a design choice made earlier for other reasons: the channel carries only a conversation id, never content. The refetch it triggers goes through the normal authorized queries, which deny a non-member. A revoked member learns that something happened in a conversation they were just removed from, never what. That turns a content leak into a metadata one.

The options, in cost order: accept it and document it, bound the stream lifetime so authorization is re-evaluated on a cadence, or publish a membership-changed signal that forces affected clients to re-subscribe. Whichever is chosen should be a decision before launch, because “authorized at subscribe” reads as continuous to anyone who has not tested it.

the finding that mattered was not about subscriptions

During a rolling restart of the subgraph, a client mutation hung for 31 seconds and then failed. My first hypothesis was drain truncation: the pod was being killed before in-flight requests finished. Raising the termination grace period to 45 seconds changed nothing. Hypothesis refuted.

The cause was the router’s transport dial timeout, which defaults to 30 seconds. During a rollout the router keeps trying to dial a pod IP that has already gone, and every request routed to it waits the full timeout before failing. Measured over 180 seconds with one write per second: the largest gap between successful writes was 32.1 seconds with one failure at the default, against 1.1 seconds with zero failures at a 3-second dial timeout and a 20-second request timeout.

Neither of the router’s configuration overlays set any traffic-shaping values. Production ran several subgraph rollouts a day across the whole graph, and the logs showed 359 subgraph fetch failures in seven days, clustering right after each deploy, including a burst of six in one second across six hosts.

Every client of the graph was exposed to this, on every deploy of any subgraph, and had been for as long as the router had been in place. Two lines of config. The recommendation separated it from the subscription decision and put it first, because bundling a production fix into an unshipped feature hides it behind a product decision that might go the other way.

what the spike was for

The feature it was investigating is still not built. The volume does not justify it yet, and a working prototype on a branch is worth more as an option than as a running service. The spike paid for itself anyway, with a withdrawn workstream, a corrected capacity model, a documented authorization gap, and a router default that had been stalling every deploy.

Measure before you extrapolate. Then check whether the thing you found is bigger than the thing you were looking for.