cm_whatsapp_bot_v1/docs/superpowers/specs/2026-05-10-windowed-fanout-design.md

Windowed, Pacing-Safe Reminder Fan-Out

Design spec for a faster, ban-safe, multi-account-friendly reminder delivery loop. Written 2026-05-10. Implementation tracked in a follow-up plan doc.

Goal

Deliver a reminder to many groups (target: 1000+) safely within a per-reminder delivery window. If we cannot finish in the window, pause the run at window-close, persist progress, and let the operator resume later from the Activity / detail view. The paused-status message tells the operator what's blocking throughput (account at capacity, media size eating the budget) so they can decide whether to offload to another paired account, shrink the attachment, or just resume the next morning at 6am.

Constraints

• WhatsApp's anti-spam is the dominant ceiling. For an established account (years of legit history), 30–60 sends/minute is the sustainable safe band; tighter for newer accounts.
• The system runs on a single bot process talking to multiple paired WhatsApp accounts. Each account's Baileys socket is independent.
• Two simultaneous fan-outs on the same WhatsApp account would double its effective send rate and risk a ban.
• The operator dropped multi-account fan-out (one reminder splitting across N accounts) earlier this week. We respect that decision — this design does not automatically split work across accounts.

Approach (selected: B)

A. Minimal pacing fix. Drop the rigid 1.5s sleep, add a token-bucket rate limiter, add window-end check, cache DB lookups. Wins ≈30% on text-only reminders; very little on media-heavy ones.

B. Pacing + media-upload cache + bounded concurrency. Everything in A, plus upload each unique media file ONCE per run via Baileys' prepareWAMessageMedia and reuse the resulting WAMediaUpload payload for every group send. Run up to N groups in parallel within one account (parts within a group stay serial so order is preserved). Wins are massive on text + picture: 1000 groups × 5 MB = 5 GB of upload turns into 5 MB. Recommended.

C. Multi-account fan-out — dropped per operator decision.

Per-account isolation (cross-account parallelism)

Today boss.work() is called with default teamSize=1, so a single fan-out monopolises the whole bot. Two reminders on different accounts queue serially, which surprises the operator.

The new model is per-account serialization, cross-account parallelism:

• teamSize raised so multiple reminders on different accounts run simultaneously.
• A per-key async mutex keyed by accountId wraps the inner work, so two reminders on the same account take turns.
• The token-bucket rate limiter is per-account too, so one account's pacing budget never throttles another.

pg-boss worker pool (teamSize = BOT_FIRE_CONCURRENCY, default 8)
  ├─ R1 (account A) ──┐
  │                    ├─ per-account-A mutex ──→ serialised within A
  ├─ R3 (account A) ──┘
  │
  ├─ R2 (account B) ──── per-account-B mutex ──→ parallel with A's
  │
  └─ R4 (account C) ──── per-account-C mutex ──→ parallel with A and B

Delivery window

Each reminder gets a window in its operator timezone. If the run cannot finish inside the window, send what we can and stop.

• New columns on reminders:
  • delivery_window_start_hour int default 6
  • delivery_window_end_hour int default 18
  • Both interpreted in the row's existing timezone column.
• Validation: 0 ≤ start < end ≤ 24. Cross-midnight windows (e.g. 22 → 06) are rejected in v1 to keep the math obvious; can be added later if anyone needs them.
• UI uses two number inputs in the When step (and edit-when page).
• delivery-window.ts exports a pure helper: windowEndAt(timezone, endHour, fireAt) → Date. Returns the end-of-window timestamp for the calendar day fireAt falls on, in the given timezone. If fireAt is already past that day's end-hour, the returned timestamp is in the past — the run loop's first iteration sees now() >= windowEndAt, marks every target skipped, and the run resolves to failed (zero sent). That's the right behaviour: "we can't send after window close, even one message".
• Only the end hour is enforced at runtime in v1. The start hour is documented on the row but not gated — operators schedule fire times that fall in their band naturally (cron + the picker's default 09:00 time fields land inside 06–18). Enforcing the start too would mean holding messages from a 4am cron miss-fire until 6am, which is a v2 conversation.

Estimated finish time (ETA preview)

The operator sets the delivery window, but they need a feel for whether the window is enough. We surface an ETA at compose and edit time so they can widen the window (or shrink the run) before hitting Schedule.

• A pure helper estimateRunDuration({ targetCount, ratePerMinute }) returns { durationMinutes, estimatedFinishAt } given a fire time. Calculation: ceil(targetCount / ratePerMinute) minutes plus a 15% buffer for per-group setup latency, with a 1-minute floor.
• Default ratePerMinute reads BOT_MAX_SEND_PER_MINUTE (40); the number is hard-coded into the web bundle as a constant — operators who tune the bot env are responsible for redeploying web. The web side does NOT read bot env directly.
• Displayed in two places:
  • Wizard Review step, between the recipients summary and the Schedule button: "~28 minutes · finishes ~10:32 (Asia/Kuala_Lumpur)"
  • Edit Groups and Edit When pages, near the save button.
• Style:
  • Green pill "Fits in window" when estimatedFinishAt <= windowEndAt.
  • Amber pill "Likely to pause" when it doesn't, with a one-line suggestion: "Widen the window or split into smaller runs."
• The ETA is advisory, not a hard gate — the operator can still schedule a run that's likely to pause; pause-and-resume covers that case. The ETA just removes the surprise.

Run loop changes (fire-reminder.ts)

Up-front, once per run:

1. Load all reminder.targets, reminder.messages, and referenced media_files rows into in-memory Maps. Drops ~3000 round-trips to ~3 round-trips for a 1000-group run.
2. Pre-create every reminder_run_targets row with status = "pending" so progress is observable from the Activity tab while the fan-out is mid-flight.
3. Pre-upload each unique media via Baileys' prepareWAMessageMedia. Cache the resulting WAMediaUpload payload keyed by mediaId for the duration of the run.
4. Compute windowEndAt and stash it.

Per-target (limited to BOT_GROUP_CONCURRENCY parallel groups, default 3):

1. Window-end gate: if Date.now() >= windowEndAt, mark the target skipped with error="delivery window closed" and skip.
2. Already-sent gate: if the run-target row is already sent (i.e. a retry is replaying), skip.
3. Acquire a token from the per-account rate limiter (default 40 msg/min, configurable BOT_MAX_SEND_PER_MINUTE).
4. assertSessions(group) — call once per group, cache for the run.
5. For each part in reminder.messages:
   • text → socket.sendMessage(jid, { text })
   • media → socket.sendMessage(jid, uploadedMediaCache[mediaId])
   • sleep jitter(200..500 ms) between parts (replaces the rigid 1.5 s wait — preserves per-chat ordering at WA's natural pace).
6. Update the run-target row to sent with latency.

Final status:

• success — every target sent.
• paused — window closed mid-run with at least one target still in pending. Run carries a resumable state: sent rows stay sent, unstarted rows stay pending (NOT skipped), failed rows stay failed. error_summary reads: "Delivery window closed at 18:00 (Asia/Kuala_Lumpur). 412 of 1000 groups delivered, 588 still pending. Resume from the Activity tab. If this happens repeatedly, consider offloading to another paired account, or shrinking the message body / media size to fit more groups in your daily window."
• partial — every target was attempted; some sent and some failed/skipped (group missing from DB, account offline, send error inside the window). Not resumable; the failures are real failures.
• failed — zero sent. Either every send errored, or the run hit the window close BEFORE the first send (run fired too late to do any work; nothing to resume).

Resume action

A paused run can be resumed by the operator. Mechanism:

• New server action resumeReminderRunAction(runId) validates ownership, then enqueues a pg-boss job: boss.send("reminder.fire", { reminderId, runId }) with NO singletonKey (resumes don't conflict with the reminder's normal cron firing).
• The fire-reminder handler accepts an optional runId in its payload. When present, it ATTACHES to that run instead of creating a new one:
  • Skips creating a new reminder_runs row.
  • Loads the existing run's reminder_run_targets rows.
  • Iterates only those with status = 'pending'.
  • Re-uses the same windowEnd / rate limiter / media cache logic as a fresh fire.
  • On window close again, status flips back to paused with an updated count.
  • On success this round, status becomes success (if no failures accumulated) or partial (if some failed).
• failed targets from the previous run are NOT retried on resume. They're real errors — surfacing them as actionable in the UI is a v2 concern (manual "retry failed" button).

UI surfaces of paused runs:

• Activity tab gets an amber "Paused" pill alongside the existing Success/Partial/Failed/Skipped/Archived filters. Resume button inline on each paused row.

• Reminder detail page's run history shows the same Resume button on paused rows. A prominent banner at the top of the detail page surfaces the latest paused run with two buttons side-by-side: Resume (re-enqueues the run via the action above) and Cancel run (marks the run partial so it stops appearing as paused; pending targets flip to skipped with error="canceled by operator"). The banner is the operator's "interactive" resume/cancel choice referenced from the pause notification.

• The reminder.fired SSE event for status=paused triggers a notification with title "Reminder paused" and body "X of Y groups delivered. Tap to resume or cancel." Clicking the notification deep-links to the detail page where the banner lives.

  Note on the Notifications API: page-side new Notification() does not support inline action buttons (only service-worker push notifications do). The "interactive" choice is therefore one click into the detail page — fewer surfaces to keep in sync, no service worker required.

Notification body

The existing reminder.fired SSE event already carries { status }. We extend it to carry sent and total counts so the notification can be specific. The notification mapper:

• success → unchanged.
• partial → body mentions delivered/total counts when present.
• paused → headline "Reminder paused", body "X of Y groups delivered. Tap to resume or cancel." Click takes the operator to the reminder's detail page where the Resume / Cancel banner lives.
• failed → unchanged.
• skipped → still filtered (bookkeeping noise).

Components

File	Role	LOC est.
migrations/0008_*.sql	add 2 int columns to reminders	<20
packages/db/src/schema.ts	drizzle alignment	<10
apps/bot/src/scheduler/per-key-mutex.ts (new)	accountId-keyed async mutex	~40
apps/bot/src/scheduler/rate-limiter.ts (new)	per-account token bucket	~60
apps/bot/src/scheduler/media-upload-cache.ts (new)	prepareWAMessageMedia results, keyed by mediaId	~50
apps/bot/src/scheduler/delivery-window.ts (new)	pure window-end calculator	~30
apps/bot/src/scheduler/fire-reminder.ts (rewrite)	new loop using all of the above	~220
apps/bot/src/scheduler/reminder-jobs.ts	teamSize config	<10
apps/bot/src/env.ts	BOT_FIRE_CONCURRENCY, BOT_MAX_SEND_PER_MINUTE, BOT_GROUP_CONCURRENCY	<20
apps/web/src/actions/reminders.ts	accept the two new fields + resumeReminderRunAction + cancelReminderRunAction	~80
apps/web/src/components/reminder-wizard/when-form-client.tsx	"Delivery hours" inputs	<40
apps/web/src/components/reminder-edit/edit-when-form.tsx	same	<30
apps/web/src/lib/run-eta.ts (new)	pure ETA calculator	~40
apps/web/src/components/reminder-wizard/run-eta-pill.tsx (new)	shared green/amber pill component	~50
apps/web/src/components/reminder-detail/paused-run-banner.tsx (new)	"Resume / Cancel run" banner	~70
apps/web/src/lib/notifications.ts	paused + partial body extension	<30

Tests

• delivery-window.test.ts — pure function. Window in past → next-day end; window crosses midnight (start > end) — explicitly reject in the schema; timezone offsets handled correctly.
• rate-limiter.test.ts — fake-clock token bucket. N tokens drained, then refill rate; backpressure via acquire() returning a Promise.
• per-key-mutex.test.ts — different keys do NOT block each other (parallelism); same key DOES (serialisation); a throwing handler releases the lock; cleanup removes empty entries.
• media-upload-cache.test.ts — mock socket: prepare called once per unique mediaId regardless of how many groups consume it.
• fire-reminder.test.ts (extend) — window-end gate marks remaining targets skipped (failed-from-the-start path) or leaves them pending and resolves the run paused when at least one send succeeded; resume re-attaches and only re-attempts pending rows.
• run-eta.test.ts — pure ETA helper: 1000 groups @ 40/min returns ~29 minutes (with the 15% buffer), edge cases (0 groups → 0, rate=0 → throws, fractional minutes → rounded up).
• notifications.test.ts (extend) — paused body reads "X of Y groups delivered. Tap to resume or cancel."; partial body uses sent/total when present.
• paused-run-banner.test.tsx — banner only renders when the latest run's status is paused; Resume click triggers the action; Cancel click triggers the cancel action.

Tuning knobs (env)

Var	Default	Effect
BOT_FIRE_CONCURRENCY	8	pg-boss worker pool size; max accounts running simultaneously
BOT_GROUP_CONCURRENCY	3	per-account parallel group sends
BOT_MAX_SEND_PER_MINUTE	40	per-account token-bucket rate; loosen to 60 if no flags after weeks of running, tighten to 20 if any rate-limit response

Per-reminder delivery_window_start_hour / delivery_window_end_hour default to 6/18 and can be widened (e.g. 0/24) for a specific big run.

Out of scope (v2 candidates)

• Crash resumability across bot restarts. If the bot dies mid-fan-out (mid-window), pg-boss will retry the job; the loop's pre-loaded pending rows still pick up correctly, but the in-memory rate-limiter and upload-cache state are gone — the retry re-uploads media and starts pacing from a full bucket. The paused-state resumability covered above is a different mechanism: it handles the "window closed cleanly" case end-to-end. The "bot crashed mid-window" case is degraded but not broken.
• Auto-resume next morning when window opens again (today the operator clicks Resume manually).
• Pause-by-operator (only window-close pauses; user-triggered pause mid-fan-out isn't wired).
• Retry-failed-targets action (paused-resume only re-attempts pending rows; failed rows stay failed).
• Native push action buttons (would require a service worker + push endpoint; v1 keeps the resume/cancel choice on the detail page, one click away from the notification).
• Adaptive ETA from observed rate (today the ETA uses the configured BOT_MAX_SEND_PER_MINUTE; a v2 could feed back the actual sustained rate from prior runs).
• Multi-account auto-split of a single reminder.
• Adaptive rate limiting (auto-back-off on WA rate-limit response codes; today the operator tunes the env var).

Acceptance

• 1000-group reminder with one image, established account: completes in roughly 30–50 minutes, comfortably inside a 6am–6pm window.
• Wizard Review shows ETA pill before submit. Setting an end hour that won't fit flips the pill amber and surfaces the "Likely to pause" hint; the operator can still proceed.
• Two reminders on different accounts firing within seconds of each other: both progress simultaneously, neither blocks the other.
• A run that hits the window end mid-fan-out: stops cleanly, marks the run paused, leaves un-started targets as pending, surfaces the paused-status notification with delivered/total counts.
• The detail page surfaces a Resume / Cancel banner for the paused run. Resume re-enqueues; if it pauses again, the banner re-appears with an updated count. Cancel run flips remaining targets to skipped and resolves the run partial; banner disappears.
• A run that hits the window end BEFORE any send (fired too late): resolves failed, no resume offered.
• 355 existing tests still pass; ≈40 new tests cover the new helpers, the paused/resume flow, the ETA preview, and the banner.