The requirement sounds like a setTimeout. When a chat message arrives, wait fifteen minutes for a consumer or thirty for an agent, and if the recipient still has not read it, send an email. If they read it in time, send nothing. Push notifications are fire-and-forget with no delivery receipt, so “push did not reach them” is implemented as “still unread when the timer fires”.
The interesting part is doing that reliably with several API pods, a worker deployment, and a message burst.
what was ruled out
Four candidate mechanisms existed in the codebase. Each was rejected for a specific reason.
A managed queue’s built-in delay. The precedent used for another email feature. Its delay is capped at fifteen minutes, so the thirty-minute agent timer does not fit, and there is no cancel.
A table plus a polling cron. New table, new poller, new failure modes, nothing to reuse.
An in-process timer. One already existed for a seconds-scale push batching window. It is per-process. A pod restart drops the timer, and with more than one API pod the read event that should cancel the timer can land on a different process than the one holding it. Fine for seconds, wrong for a thirty-minute window.
The raw job queue. It supports a delay and a dedup key. But nothing in the repo removed a job from it, so there was no cancel path. Its producer module was gated to consumer pods while the chat send path runs on API pods. The consumer dispatch also needed a new job-type case plus an analysis of how retention interacts with a keyed re-enqueue.
the one that was chosen, and the decision it reversed
The team’s job SDK, built on the same queue library, had been rejected in the parent ticket on the grounds that it did not support delay and dedup. I read its source. It supports both, per enqueue. It also has the two things the raw queue lacked: a real cancel (getJob then remove), and timezone-native cron scheduling with a live precedent already running in production. Its producer was already registered on API pods, used by two other features from the request path.
So the decision reversed, on the strength of reading the library rather than trusting an inherited constraint. That is worth stating on its own: the parent ticket’s rejection was reasonable given what its author believed the SDK could do, and it was wrong.
the design
On message send, after the transaction commits, the same hook that computes push recipients enqueues one delayed job per recipient. The job’s dedup key is:
unread-email:{conversationId}:{recipientId}:{messageId}On read, the read-marking path calls remove for that recipient’s pending jobs in that conversation. The author’s own read watermark advances on send, so the sender never gets a job for their own message.
At fire time, the worker re-runs the unread predicate against the recipient’s current read watermark. If they have read since, it skips silently. If not, it counts the unread messages now and sends one email with that count.
why cancel alone is not enough
The remove call swallows one race: a remove that lands exactly as the job is becoming active. The library’s own catch block absorbs that, and the job runs. Without the fire-time recheck, a recipient who read the message in the last second before the timer would get an email saying they had an unread message.
The recheck also covers a remove that never landed at all, for any reason: a pod dying between the read-mark and the remove, a transient queue error, a bug. Cancel is the first layer. The recheck is the second, and it is the one that makes the system correct rather than usually correct.
why the message id is in the key
The obvious dedup key is {conversationId}:{recipientId}: one pending job per recipient per conversation. That breaks in a specific way. A job completes, the recipient still does not read, a new message arrives, and the new enqueue is deduplicated against the completed job’s retained record. The new message never schedules a timer.
Including the message id keeps a genuinely new message always schedulable. Aggregation comes from the other two mechanisms instead. A burst of five messages in ten minutes schedules five jobs. Each read-and-reschedule cycle removes the pending ones. The job that survives to fire time counts all five as unread and sends one email saying “5 new messages”. The count is computed at fire time, never carried in the job payload from when it was scheduled, which is what makes the coalescing correct.
the daily digest, same tail
A daily digest of unread conversations uses the same send path with a different trigger: a scheduled job at 9 a.m. Pacific, registered with a native timezone rather than a UTC cron with hand-written daylight-saving adjustment. The precedent for that already existed in the codebase, so the design copied it field for field rather than inventing a new one. The Kubernetes CronJob that had been planned for it, with its own UTC schedule and daylight-saving drift, became unnecessary.
what it costs
The fallback job is the first delayed job on that system. The existing registrations are immediate or cron-scheduled. A thirty-minute delayed job exercises the queue’s retention of delayed state and the worker deployment’s uptime across the delay. Both are fine in principle. It is still the first resident of that regime, and it got a staging soak behind its own flag before production.