The biggest thing I have led is a replacement. A home-search product had a legacy feature where a buyer saved listings to a private list and shared it with their agent. Each list had one owner. The company wanted a shared workspace instead: a buyer and their agent, and later anyone else the buyer invites, saving, sharing, and discussing listings in the same place. The old feature was built on infrastructure nobody wanted to extend, and the data model had the wrong shape for what came next. A single-owner list does not become a multi-member workspace by adding columns.
The constraint that shaped everything: agents and their clients used the old feature every day, across a web app and two mobile platforms, each shipped as a separate per-tenant build, and none of them could be asked to do anything. No flag day. No forced app update. No email telling agents to migrate their lists. The transition ended up running almost two years, and for all of that time the old and new systems coexisted.
phases that ship on their own
The cutover was sequenced so that every phase was independently shippable and left the product working. Roughly:
- The new data model and backend, behind the federated graph, with nothing reading it yet.
- New write paths, dual-writing where a legacy reader still needed the old rows.
- The new surfaces on web, then mobile, gated per tenant.
- The backfill, moving historical data into the new model.
- Readers switched to the new model, one at a time.
- The legacy writers turned off, then the legacy tables dropped.
The ordering matters because each step is reversible on its own. If phase 3 had shipped a bug on mobile, phase 2 was still writing both shapes and nothing was lost. The alternative, a big-bang migration with a maintenance window, would have been faster to build and would have required exactly the coordination we were not allowed to ask for.
sixteen batches, one hex character each
Tenant IDs are UUIDs. A UUID is hexadecimal, so the first character of any tenant’s ID is one of sixteen values, and the distribution across them is close to uniform. The backfill ran in sixteen batches, one per leading hex character.
Each batch was a bounded slice of the tenant population. Bounded means we knew before running it how many tenants and roughly how many rows it covered. Verifiable means that after it ran, a per-row comparison between old and new could be scoped to exactly those tenants. Resumable means a failure in batch 7 left batches 0 to 6 done and correct, and batch 7 could be re-run on its own.
I would pick this partitioning again. It needs no bookkeeping table, no cursor, and no coordination with anything else in the system. The partition key is already on every row.
two write paths, one read model
The hardest part of coexistence is not writing to both systems. It is reading correctly while some tenants are on the old path and some are on the new one.
The mistake to avoid is a per-tenant mode lookup on every read: check which system this tenant is on, then query that one. It works until a reader is added that forgets the check, or a tenant is mid-migration and has rows in both.
What we did instead: every reader that needed recipients of a shared item walked both paths and took the union. A tenant on the legacy path has no rows in the new tables and a tenant on the new path has no rows in the legacy tables, so the union is correct for both without knowing which is which. When the legacy writer was finally turned off, the legacy leg of each union was deleted outright.
A production bug surfaced this. A share made through the new path reached nobody through mobile push or the consumer feed, because four separate readers resolved recipients by walking a legacy table the new path deliberately never wrote to. One of the four had been patched earlier through an event re-emit, which is exactly why the other three were easy to miss. The union fixed all four with the same shape.
While fixing those readers we found three unbounded list queries feeding a cron that had to see every matching row. Our own new lookup had the same defect in a different form: a 2,000-row cap plus a warning log. A cap on a read that must be complete trades an eventual memory problem for immediate, silent, missing notifications. All of them page by cursor now.
the migration window needs “both off”
Late in the transition the new surface itself got replaced by a further evolution of it, and the same problem came up in a smaller form. Two surfaces, one predicate deciding which a tenant sees, and a window during the rollout where a tenant should see neither.
One flag cannot express “both off”. The first attempt gated the new surface on an enable flag and left the old surface gated on the entitlement alone, so a tenant with the new one enabled saw both menu entries and could open both, each rendering a different half of the same data.
The fix was a second flag with inverted polarity, a kill switch for the old surface that defaults to false, so a flag the client cannot resolve fails toward today’s behavior, plus one pure function that returns which surfaces are visible. The polarity decision has its own post: name your feature flag for what happens when it cannot be read.
what it looked like from outside
Nothing. An agent who used the old feature on a Monday used the new one on a Tuesday because their tenant had been switched, with their historical data already there. No one was asked to migrate anything. That is the whole claim, and the sequencing above is the proof of it.
The replacement is now the foundation for real-time chat between clients and their agent, which the legacy model could never have supported. The migration and the chat are one arc: replace the thing cleanly, then build on it what the old thing could not carry.