Inbound leads, home buyers and sellers filling in a form on a brokerage website, used to land in the CRM unassigned. Every hour a lead sat without an agent, the odds of converting it dropped. Speed to contact is the biggest single lever on inbound conversion, so the job was to get each prospect to the right agent quickly and automatically. I built the routing engine in 2023 and still own it. This is the part of it that interviewers push on hardest: what happens when two leads arrive at once.

routing, then round-robin

The first version was rules. A lead has attributes (source, location, price range, the form it came from), a tenant has agents and teams, and a set of rules maps one to the other. That gets a lead to an agent in one hop and covers most tenants.

The extension was fair distribution. A team wants inbound leads spread across its agents in turn instead of all going to whoever the first matching rule names. That is round-robin, and round-robin is where the correctness problems live, because it has state: whose turn it is.

Three pieces sit on top of the rotation:

• Rule precedence. Specific rules beat general ones. A rule for “buyer leads over a price threshold from this campaign” wins over “all buyer leads”, which wins over the tenant default. Precedence is explicit and ordered, never inferred from creation date.
• Priority. Within a matching set, a rule can rank agents so a rotation is weighted rather than flat.
• Skip logic. An agent who is unavailable, or already assigned by a higher-precedence rule for this lead, is skipped without consuming the turn. Skipping is the operation most likely to corrupt the rotation if the state is not handled carefully, because a skip that advances the marker and a skip that does not produce different fairness over time.

the concurrency problem

Two leads arrive for the same team within the same few milliseconds. Both handlers read “next agent is B”. Both assign to B. Both advance the marker to C. Agent B has two leads, agent C’s turn was consumed by nobody, and the rotation is now permanently shifted by one. Nothing crashes. The bug is a fairness drift that shows up weeks later as an agent complaining they get fewer leads than a colleague, and it cannot be reproduced on demand.

Anything that reads the rotation state, decides, and writes it back without holding the state in between has this bug.

a row lock over a persisted position marker

The rotation state is a row: one per round-robin queue, holding the current position. Assignment runs in a transaction that locks that row first (SELECT ... FOR UPDATE), then reads the position, applies the skip logic, writes the assignment, advances the marker, and commits.

Concurrent arrivals serialize on the lock. The second handler blocks until the first commits, then reads the position the first one wrote. Each advance is atomic with the assignment it belongs to, so there is no window where a lead is assigned but the marker has not moved, or the marker has moved but the lead is not assigned. The skip logic reads and writes the same locked state, so a skip is as consistent as an assignment.

The question that follows is what a lock buys over an optimistic or queue-based design. Assignment is low-contention: a team’s leads arrive seconds or minutes apart most of the time, with occasional bursts. But it must be exactly-once. For that shape, a short row lock is the simplest mechanism that is provably correct. Optimistic concurrency (read a version, write with a version check, retry on conflict) is correct too, and it is what I would reach for under high contention, but it moves the retry loop into application code where every skip rule has to be re-run and the failure mode is a retry storm during exactly the bursts that matter. A queue serializes for free but adds a component, a consumer, and a delay to a path whose whole point is speed. The lock is held for one short transaction on one row per team. It has never been the bottleneck.

shipping a stateful engine in reversible stages

The rotation state, the precedence rules, and the skip logic each changed the schema. Each stage rolled out behind its own feature flag with its own migration, so every step could be observed in production before the next one started and turned off without a rollback of the code. The flag order matters: the migration adds the column or table, the flag turns on the read path, and only when the read path is proven does the flag turn on the write path that depends on it.

That is slower than shipping the engine in one release. It is also the reason a system that decides who gets revenue has never needed an emergency rollback.

what to test

The concurrent-arrival case gets a real test: two transactions started against the same queue row, asserted to produce two distinct agents and a marker advanced by exactly two. Rule precedence gets a table-driven test where every row is a lead shape and the expected winning rule. Skip logic gets the case where the next agent in rotation is unavailable and the assertion is both who got the lead and where the marker ended up. The last one is the test that catches the drift bug, because it pins down whether a skip consumes a turn.