cm_whatsapp_bot_v1/docs/superpowers/specs/2026-05-10-windowed-fanout-design.md

Windowed, Pacing-Safe Reminder Fan-Out

Design spec for a faster, ban-safe, multi-account-friendly reminder delivery loop. Written 2026-05-10. Implementation tracked in a follow-up plan doc.

Goal

Deliver a reminder to many groups (target: 1000+) safely within a per-reminder delivery window. If we cannot finish in the window, stop, mark the run partial, and tell the operator the account is at capacity for this fan-out.

Constraints

• WhatsApp's anti-spam is the dominant ceiling. For an established account (years of legit history), 30–60 sends/minute is the sustainable safe band; tighter for newer accounts.
• The system runs on a single bot process talking to multiple paired WhatsApp accounts. Each account's Baileys socket is independent.
• Two simultaneous fan-outs on the same WhatsApp account would double its effective send rate and risk a ban.
• The operator dropped multi-account fan-out (one reminder splitting across N accounts) earlier this week. We respect that decision — this design does not automatically split work across accounts.

Approach (selected: B)

A. Minimal pacing fix. Drop the rigid 1.5s sleep, add a token-bucket rate limiter, add window-end check, cache DB lookups. Wins ≈30% on text-only reminders; very little on media-heavy ones.

B. Pacing + media-upload cache + bounded concurrency. Everything in A, plus upload each unique media file ONCE per run via Baileys' prepareWAMessageMedia and reuse the resulting WAMediaUpload payload for every group send. Run up to N groups in parallel within one account (parts within a group stay serial so order is preserved). Wins are massive on text + picture: 1000 groups × 5 MB = 5 GB of upload turns into 5 MB. Recommended.

C. Multi-account fan-out — dropped per operator decision.

Per-account isolation (cross-account parallelism)

Today boss.work() is called with default teamSize=1, so a single fan-out monopolises the whole bot. Two reminders on different accounts queue serially, which surprises the operator.

The new model is per-account serialization, cross-account parallelism:

• teamSize raised so multiple reminders on different accounts run simultaneously.
• A per-key async mutex keyed by accountId wraps the inner work, so two reminders on the same account take turns.
• The token-bucket rate limiter is per-account too, so one account's pacing budget never throttles another.

pg-boss worker pool (teamSize = BOT_FIRE_CONCURRENCY, default 8)
  ├─ R1 (account A) ──┐
  │                    ├─ per-account-A mutex ──→ serialised within A
  ├─ R3 (account A) ──┘
  │
  ├─ R2 (account B) ──── per-account-B mutex ──→ parallel with A's
  │
  └─ R4 (account C) ──── per-account-C mutex ──→ parallel with A and B

Delivery window

Each reminder gets a window in its operator timezone. If the run cannot finish inside the window, send what we can and stop.

• New columns on reminders:
  • delivery_window_start_hour int default 6
  • delivery_window_end_hour int default 18
  • Both interpreted in the row's existing timezone column.
• Validation: 0 ≤ start < end ≤ 24. Cross-midnight windows (e.g. 22 → 06) are rejected in v1 to keep the math obvious; can be added later if anyone needs them.
• UI uses two number inputs in the When step (and edit-when page).
• delivery-window.ts exports a pure helper: windowEndAt(timezone, endHour, fireAt) → Date. Returns the end-of-window timestamp for the calendar day fireAt falls on, in the given timezone. If fireAt is already past that day's end-hour, the returned timestamp is in the past — the run loop's first iteration sees now() >= windowEndAt, marks every target skipped, and the run resolves to failed (zero sent). That's the right behaviour: "we can't send after window close, even one message".
• Only the end hour is enforced at runtime in v1. The start hour is documented on the row but not gated — operators schedule fire times that fall in their band naturally (cron + the picker's default 09:00 time fields land inside 06–18). Enforcing the start too would mean holding messages from a 4am cron miss-fire until 6am, which is a v2 conversation.

Run loop changes (fire-reminder.ts)

Up-front, once per run:

1. Load all reminder.targets, reminder.messages, and referenced media_files rows into in-memory Maps. Drops ~3000 round-trips to ~3 round-trips for a 1000-group run.
2. Pre-create every reminder_run_targets row with status = "pending" so progress is observable from the Activity tab while the fan-out is mid-flight.
3. Pre-upload each unique media via Baileys' prepareWAMessageMedia. Cache the resulting WAMediaUpload payload keyed by mediaId for the duration of the run.
4. Compute windowEndAt and stash it.

Per-target (limited to BOT_GROUP_CONCURRENCY parallel groups, default 3):

1. Window-end gate: if Date.now() >= windowEndAt, mark the target skipped with error="delivery window closed" and skip.
2. Already-sent gate: if the run-target row is already sent (i.e. a retry is replaying), skip.
3. Acquire a token from the per-account rate limiter (default 40 msg/min, configurable BOT_MAX_SEND_PER_MINUTE).
4. assertSessions(group) — call once per group, cache for the run.
5. For each part in reminder.messages:
   • text → socket.sendMessage(jid, { text })
   • media → socket.sendMessage(jid, uploadedMediaCache[mediaId])
   • sleep jitter(200..500 ms) between parts (replaces the rigid 1.5 s wait — preserves per-chat ordering at WA's natural pace).
6. Update the run-target row to sent with latency.

Final status:

• success — every target sent.
• partial — at least one sent, at least one not (window-close, failed, missing group). error_summary reads: "Delivery window closed at 18:00 (Asia/Kuala_Lumpur). 412 of 1000 groups delivered. This account is at capacity for this fan-out — consider sending the remainder from another paired account."
• failed — zero sent.

Notification body

The existing reminder.fired SSE event already carries { status }. The web's notification mapper already handles partial with a "see activity" hint. The body extends to mention "X of Y delivered" when status === "partial".

Components

File	Role	LOC est.
migrations/0008_*.sql	add 2 int columns to reminders	<20
packages/db/src/schema.ts	drizzle alignment	<10
apps/bot/src/scheduler/per-key-mutex.ts (new)	accountId-keyed async mutex	~40
apps/bot/src/scheduler/rate-limiter.ts (new)	per-account token bucket	~60
apps/bot/src/scheduler/media-upload-cache.ts (new)	prepareWAMessageMedia results, keyed by mediaId	~50
apps/bot/src/scheduler/delivery-window.ts (new)	pure window-end calculator	~30
apps/bot/src/scheduler/fire-reminder.ts (rewrite)	new loop using all of the above	~200
apps/bot/src/scheduler/reminder-jobs.ts	teamSize config	<10
apps/bot/src/env.ts	BOT_FIRE_CONCURRENCY, BOT_MAX_SEND_PER_MINUTE, BOT_GROUP_CONCURRENCY	<20
apps/web/src/actions/reminders.ts	accept the two new fields	<30
apps/web/src/components/reminder-wizard/when-form-client.tsx	"Delivery hours" inputs	<40
apps/web/src/components/reminder-edit/edit-when-form.tsx	same	<30
apps/web/src/lib/notifications.ts	partial-status body extension	<15

Tests

• delivery-window.test.ts — pure function. Window in past → next-day end; window crosses midnight (start > end) — explicitly reject in the schema; timezone offsets handled correctly.
• rate-limiter.test.ts — fake-clock token bucket. N tokens drained, then refill rate; backpressure via acquire() returning a Promise.
• per-key-mutex.test.ts — different keys do NOT block each other (parallelism); same key DOES (serialisation); a throwing handler releases the lock; cleanup removes empty entries.
• media-upload-cache.test.ts — mock socket: prepare called once per unique mediaId regardless of how many groups consume it.
• fire-reminder.test.ts (extend) — window-end gate marks remaining targets skipped; partial-status error_summary includes account / delivered / total context.

Tuning knobs (env)

Var	Default	Effect
BOT_FIRE_CONCURRENCY	8	pg-boss worker pool size; max accounts running simultaneously
BOT_GROUP_CONCURRENCY	3	per-account parallel group sends
BOT_MAX_SEND_PER_MINUTE	40	per-account token-bucket rate; loosen to 60 if no flags after weeks of running, tighten to 20 if any rate-limit response

Per-reminder delivery_window_start_hour / delivery_window_end_hour default to 6/18 and can be widened (e.g. 0/24) for a specific big run.

Out of scope (v2 candidates)

• Crash resumability across bot restarts. Today, if the bot dies mid-fan-out, pg-boss will retry the job; the loop will skip any rows already marked sent, but the in-memory rate-limiter and upload-cache state are gone — meaning the retry uploads media again and starts pacing from a full bucket. Acceptable for v1.
• Pause / resume mid-run controls.
• Cross-day window resume (current design hard-stops at window end and reports partial; doesn't queue the remainder for tomorrow).
• Multi-account auto-split of a single reminder.
• Adaptive rate limiting (auto-back-off on WA rate-limit response codes; today the operator tunes the env var).

Acceptance

• 1000-group reminder with one image, established account: completes in roughly 30–50 minutes, comfortably inside a 6am–6pm window.
• Two reminders on different accounts firing within seconds of each other: both progress simultaneously, neither blocks the other.
• A run that hits the window end: stops cleanly, marks remaining as skipped, surfaces the partial-status message in the Activity tab and via the browser notification.
• 355 existing tests still pass; ≈25 new tests cover the new helpers.