cm_whatsapp_bot_v1/docs/superpowers/specs/2026-05-10-windowed-fanout-design.md

# Windowed, Pacing-Safe Reminder Fan-Out

> Design spec for a faster, ban-safe, multi-account-friendly reminder
> delivery loop. Written 2026-05-10. Implementation tracked in a
> follow-up plan doc.

## Goal

Deliver a reminder to many groups (target: 1000+) safely within a
per-reminder delivery window. If we cannot finish in the window,
**pause** the run at window-close, persist progress, and let the
operator **resume** later from the Activity / detail view. The
paused-status message tells the operator what's blocking throughput
(account at capacity, media size eating the budget) so they can
decide whether to offload to another paired account, shrink the
attachment, or just resume the next morning at 6am.

## Constraints

- WhatsApp's anti-spam is the dominant ceiling. For an established
  account (years of legit history), 30–60 sends/minute is the
  sustainable safe band; tighter for newer accounts.
- The system runs on a single bot process talking to multiple paired
  WhatsApp accounts. Each account's Baileys socket is independent.
- Two simultaneous fan-outs on the **same** WhatsApp account would
  double its effective send rate and risk a ban.
- The operator dropped multi-account fan-out (one reminder splitting
  across N accounts) earlier this week. We respect that decision —
  this design does not automatically split work across accounts.

## Approach (selected: B)

**A. Minimal pacing fix.** Drop the rigid 1.5s sleep, add a
token-bucket rate limiter, add window-end check, cache DB lookups.
Wins ≈30% on text-only reminders; very little on media-heavy ones.

**B. Pacing + media-upload cache + bounded concurrency.** Everything
in A, plus upload each unique media file ONCE per run via Baileys'
`prepareWAMessageMedia` and reuse the resulting `WAMediaUpload`
payload for every group send. Run up to N groups in parallel within
one account (parts within a group stay serial so order is preserved).
Wins are massive on text + picture: 1000 groups × 5 MB = 5 GB of
upload turns into 5 MB. **Recommended.**

**C. Multi-account fan-out** — dropped per operator decision.

## Per-account isolation (cross-account parallelism)

Today `boss.work()` is called with default `teamSize=1`, so a single
fan-out monopolises the whole bot. Two reminders on **different**
accounts queue serially, which surprises the operator.

The new model is **per-account serialization, cross-account
parallelism**:

- `teamSize` raised so multiple reminders on different accounts run
  simultaneously.
- A per-key async mutex keyed by `accountId` wraps the inner work, so
  two reminders on the **same** account take turns.
- The token-bucket rate limiter is per-account too, so one account's
  pacing budget never throttles another.

```
pg-boss worker pool (teamSize = BOT_FIRE_CONCURRENCY, default 8)
  ├─ R1 (account A) ──┐
  │                    ├─ per-account-A mutex ──→ serialised within A
  ├─ R3 (account A) ──┘
  │
  ├─ R2 (account B) ──── per-account-B mutex ──→ parallel with A's
  │
  └─ R4 (account C) ──── per-account-C mutex ──→ parallel with A and B
```

## Delivery window

Each reminder gets a window in its operator timezone. If the run
cannot finish inside the window, send what we can and stop.

- New columns on `reminders`:
  - `delivery_window_start_hour int default 6`
  - `delivery_window_end_hour int default 18`
  - Both interpreted in the row's existing `timezone` column.
- Validation: `0 ≤ start < end ≤ 24`. Cross-midnight windows (e.g.
  22 → 06) are rejected in v1 to keep the math obvious; can be added
  later if anyone needs them.
- UI uses two number inputs in the When step (and edit-when page).
- `delivery-window.ts` exports a pure helper:
  `windowEndAt(timezone, endHour, fireAt) → Date`. Returns the
  end-of-window timestamp for the calendar day `fireAt` falls on,
  in the given timezone. If `fireAt` is already past that day's
  end-hour, the returned timestamp is in the past — the run loop's
  first iteration sees `now() >= windowEndAt`, marks every target
  `skipped`, and the run resolves to `failed` (zero sent). That's
  the right behaviour: "we can't send after window close, even one
  message".
- **Only the end hour is enforced at runtime in v1.** The start
  hour is documented on the row but not gated — operators schedule
  fire times that fall in their band naturally (cron + the picker's
  default 09:00 time fields land inside 06–18). Enforcing the start
  too would mean holding messages from a 4am cron miss-fire until
  6am, which is a v2 conversation.

## Run loop changes (`fire-reminder.ts`)

Up-front, once per run:

1. Load all `reminder.targets`, `reminder.messages`, and referenced
   `media_files` rows into in-memory Maps. Drops ~3000 round-trips to
   ~3 round-trips for a 1000-group run.
2. Pre-create every `reminder_run_targets` row with
   `status = "pending"` so progress is observable from the Activity
   tab while the fan-out is mid-flight.
3. **Pre-upload each unique media** via Baileys'
   `prepareWAMessageMedia`. Cache the resulting `WAMediaUpload`
   payload keyed by `mediaId` for the duration of the run.
4. Compute `windowEndAt` and stash it.

Per-target (limited to `BOT_GROUP_CONCURRENCY` parallel groups,
default 3):

1. **Window-end gate:** if `Date.now() >= windowEndAt`, mark the
   target `skipped` with `error="delivery window closed"` and skip.
2. **Already-sent gate:** if the run-target row is already `sent`
   (i.e. a retry is replaying), skip.
3. Acquire a token from the per-account rate limiter (default 40
   msg/min, configurable `BOT_MAX_SEND_PER_MINUTE`).
4. `assertSessions(group)` — call once per group, cache for the run.
5. For each part in `reminder.messages`:
   - text → `socket.sendMessage(jid, { text })`
   - media → `socket.sendMessage(jid, uploadedMediaCache[mediaId])`
   - sleep `jitter(200..500 ms)` between parts (replaces the rigid
     1.5 s wait — preserves per-chat ordering at WA's natural pace).
6. Update the run-target row to `sent` with latency.

Final status:

- **success** — every target sent.
- **paused** — window closed mid-run with at least one target still in
  `pending`. Run carries a resumable state: sent rows stay `sent`,
  unstarted rows stay `pending` (NOT skipped), failed rows stay
  `failed`. `error_summary` reads:
  `"Delivery window closed at 18:00 (Asia/Kuala_Lumpur). 412 of 1000
  groups delivered, 588 still pending. Resume from the Activity tab.
  If this happens repeatedly, consider offloading to another paired
  account, or shrinking the message body / media size to fit more
  groups in your daily window."`
- **partial** — every target was attempted; some sent and some
  failed/skipped (group missing from DB, account offline, send error
  inside the window). Not resumable; the failures are real failures.
- **failed** — zero sent. Either every send errored, or the run hit
  the window close BEFORE the first send (run fired too late to do
  any work; nothing to resume).

## Resume action

A paused run can be resumed by the operator. Mechanism:

- New server action `resumeReminderRunAction(runId)` validates
  ownership, then enqueues a pg-boss job:
  `boss.send("reminder.fire", { reminderId, runId })` with NO
  singletonKey (resumes don't conflict with the reminder's normal
  cron firing).
- The fire-reminder handler accepts an optional `runId` in its
  payload. When present, it ATTACHES to that run instead of creating
  a new one:
  - Skips creating a new `reminder_runs` row.
  - Loads the existing run's `reminder_run_targets` rows.
  - Iterates only those with `status = 'pending'`.
  - Re-uses the same windowEnd / rate limiter / media cache logic as
    a fresh fire.
  - On window close again, status flips back to `paused` with an
    updated count.
  - On success this round, status becomes `success` (if no failures
    accumulated) or `partial` (if some failed).
- `failed` targets from the previous run are NOT retried on resume.
  They're real errors — surfacing them as actionable in the UI is a
  v2 concern (manual "retry failed" button).

UI surfaces of paused runs:

- Activity tab gets an amber "Paused" pill alongside the existing
  Success/Partial/Failed/Skipped/Archived filters. Resume button
  inline on each paused row.
- Reminder detail page's run history shows the same Resume button on
  paused rows.
- The `reminder.fired` SSE event for status=paused triggers a
  notification with title "Reminder paused" and body
  `"X of Y groups delivered. Resume from the Activity tab."`

## Notification body

The existing `reminder.fired` SSE event already carries `{ status }`.
The notification mapper extends:

- `success` → unchanged.
- `partial` → body mentions delivered/total counts when present.
- `paused` → headline `"Reminder paused"`, body
  `"X of Y groups delivered. Resume from the Activity tab."` Click
  takes the operator to the reminder's detail page where the Resume
  button lives.
- `failed` → unchanged.
- `skipped` → still filtered (bookkeeping noise).

## Components

| File | Role | LOC est. |
| --- | --- | --- |
| `migrations/0008_*.sql` | add 2 int columns to `reminders` | <20 |
| `packages/db/src/schema.ts` | drizzle alignment | <10 |
| `apps/bot/src/scheduler/per-key-mutex.ts` (new) | accountId-keyed async mutex | ~40 |
| `apps/bot/src/scheduler/rate-limiter.ts` (new) | per-account token bucket | ~60 |
| `apps/bot/src/scheduler/media-upload-cache.ts` (new) | `prepareWAMessageMedia` results, keyed by mediaId | ~50 |
| `apps/bot/src/scheduler/delivery-window.ts` (new) | pure window-end calculator | ~30 |
| `apps/bot/src/scheduler/fire-reminder.ts` (rewrite) | new loop using all of the above | ~200 |
| `apps/bot/src/scheduler/reminder-jobs.ts` | `teamSize` config | <10 |
| `apps/bot/src/env.ts` | `BOT_FIRE_CONCURRENCY`, `BOT_MAX_SEND_PER_MINUTE`, `BOT_GROUP_CONCURRENCY` | <20 |
| `apps/web/src/actions/reminders.ts` | accept the two new fields | <30 |
| `apps/web/src/components/reminder-wizard/when-form-client.tsx` | "Delivery hours" inputs | <40 |
| `apps/web/src/components/reminder-edit/edit-when-form.tsx` | same | <30 |
| `apps/web/src/lib/notifications.ts` | partial-status body extension | <15 |

## Tests

- `delivery-window.test.ts` — pure function. Window in past →
  next-day end; window crosses midnight (start > end) — explicitly
  reject in the schema; timezone offsets handled correctly.
- `rate-limiter.test.ts` — fake-clock token bucket. N tokens drained,
  then refill rate; backpressure via `acquire()` returning a Promise.
- `per-key-mutex.test.ts` — different keys do NOT block each other
  (parallelism); same key DOES (serialisation); a throwing handler
  releases the lock; cleanup removes empty entries.
- `media-upload-cache.test.ts` — mock socket: `prepare` called once
  per unique mediaId regardless of how many groups consume it.
- `fire-reminder.test.ts` (extend) — window-end gate marks remaining
  targets `skipped`; partial-status error_summary includes account /
  delivered / total context.

## Tuning knobs (env)

| Var | Default | Effect |
| --- | --- | --- |
| `BOT_FIRE_CONCURRENCY` | 8 | pg-boss worker pool size; max accounts running simultaneously |
| `BOT_GROUP_CONCURRENCY` | 3 | per-account parallel group sends |
| `BOT_MAX_SEND_PER_MINUTE` | 40 | per-account token-bucket rate; loosen to 60 if no flags after weeks of running, tighten to 20 if any rate-limit response |

Per-reminder `delivery_window_start_hour` / `delivery_window_end_hour`
default to 6/18 and can be widened (e.g. 0/24) for a specific big run.

## Out of scope (v2 candidates)

- **Crash resumability across bot restarts.** If the bot dies
  mid-fan-out (mid-window), pg-boss will retry the job; the loop's
  pre-loaded `pending` rows still pick up correctly, but the
  in-memory rate-limiter and upload-cache state are gone — the
  retry re-uploads media and starts pacing from a full bucket. The
  paused-state resumability covered above is a different mechanism:
  it handles the "window closed cleanly" case end-to-end. The
  "bot crashed mid-window" case is degraded but not broken.
- **Auto-resume next morning** when window opens again (today the
  operator clicks Resume manually).
- **Pause-by-operator** (only window-close pauses; user-triggered
  pause mid-fan-out isn't wired).
- **Retry-failed-targets** action (paused-resume only re-attempts
  `pending` rows; `failed` rows stay failed).
- **Multi-account auto-split** of a single reminder.
- **Adaptive rate limiting** (auto-back-off on WA rate-limit response
  codes; today the operator tunes the env var).

## Acceptance

- 1000-group reminder with one image, established account: completes
  in roughly 30–50 minutes, comfortably inside a 6am–6pm window.
- Two reminders on different accounts firing within seconds of each
  other: both progress simultaneously, neither blocks the other.
- A run that hits the window end mid-fan-out: stops cleanly, marks
  the run `paused`, leaves un-started targets as `pending`, surfaces
  the paused-status notification with delivered/total counts.
- The operator clicks **Resume** on a paused run — fan-out continues
  from the unsent targets, respecting the same per-account rate
  limit + window. If it again can't finish, it pauses again with an
  updated count.
- A run that hits the window end BEFORE any send (fired too late):
  resolves `failed`, no resume offered.
- 355 existing tests still pass; ≈30 new tests cover the new helpers
  and the paused/resume flow.