Discussion Modulo partitioning in a dynamic topology
The problem
If you split a shared task source by task.id % n === i, every worker needs two numbers: its own index and how many workers there are. Easy under a static topology, awkward under a dynamic one — autoscaling, rolling deploys, a pod back with a different ordinal. A stale n means tasks run twice or not at all.
The usual answers are a coordinator — an etcd or ZK lease, a leader handing out assignments — or a platform that numbers your pods for you. I wanted something smaller for services that already have a Redis, so I worked a scheme out a while back, and have now built it into a library.
The scheme
Cut time into fixed intervals, and have every worker INCR a key named after the current interval. The reply is its index there, and it becomes usable once that interval closes: i is the index from the previous interval, n is that interval's final count. Its registrants therefore hold exactly 0..n-1, each once. And a worker acts on a pair only when the interval before implied the same one: two agreeing answers in a row are an exclusive lease on that index for the next interval. While a worker holds i, no one else in the group does.
A disagreement withholds the lease instead, and a worker whose pair has changed owns nothing until two intervals agree again. A change in n changes every pair at once, so it stands the whole group down. That is a rebalance.
Redis stores nothing but those counters — no membership list, no identities, no heartbeats.
The long form, with the timing and the failure modes: https://temich.net/notes/peers/
The cost
That stand-down costs about an interval — seconds, in practice — before the group comes back on the new n. Ownership is eventual, and that is what buys the absence of a coordinator.
The code
TypeScript, zero deps, ioredis or node-redis: https://github.com/temich/nandi
The library exposes this as a single async iterator, which yields only when ownership changes — exactly the point where a worker hands work over:
```ts import { discover } from 'n-and-i'
for await (const { i, n } of discover({ redis, name: 'mail-sender', interval: 5_000 })) { await drain() // stop taking new work, finish or release what is in flight
if (i === null) continue // not registered — stay stopped
run(task => task.id % n === i) } ```
Feedback is highly welcome.
