From 94ef5595ea45e6155fb4fdf4a0eafbd110bc0484 Mon Sep 17 00:00:00 2001 From: yiekheng Date: Sat, 2 May 2026 16:49:12 +0800 Subject: [PATCH] Add design spec for local-as-dev tier (sub-project A) Adds containerized MySQL to docker-compose.override.yml, gates telegram/transfer bots behind a 'bots' profile, and introduces a local Python bot CLI with a stdlib TUI menu that mirrors Telegram's /1, /2, /3 plus operational subcommands. CLI runs from .venv against 127.0.0.1:3306 (mysql published to localhost only). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../specs/2026-05-02-local-as-dev-design.md | 499 ++++++++++++++++++ 1 file changed, 499 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-02-local-as-dev-design.md diff --git a/docs/superpowers/specs/2026-05-02-local-as-dev-design.md b/docs/superpowers/specs/2026-05-02-local-as-dev-design.md new file mode 100644 index 0000000..400c2a3 --- /dev/null +++ b/docs/superpowers/specs/2026-05-02-local-as-dev-design.md @@ -0,0 +1,499 @@ +# Local-as-Dev Tier (Sub-Project A) Design + +**Date:** 2026-05-02 +**Status:** Approved (design) +**Sequel to:** [2026-05-02-debug-mode-hotfix-design.md](2026-05-02-debug-mode-hotfix-design.md) +**Out of scope (separate):** rex/siong env file rotation (R2), cm_bot.py scraper resilience (R3), security hardening sub-project (C), Next.js web view (B). + +## Problem + +`cm_bot_v2` runs two production deployments — `rex` (port 8001) and `siong` (port 8005) — both deployed via Portainer from images on `gitea.04080616.xyz/yiekheng`. There is no formal local development tier. Local iteration today either touches a real production database or has no DB at all (`web-view` is the only service that boots without one). This blocks safe testing of the security hardening (sub-project C) and the planned Next.js rewrite (B), both of which need to be exercised end-to-end without poking prod data. + +## Goal + +Add a local-only "dev tier" that spins up `mysql` + `api-server` + `web-view` self-contained on a developer machine, plus a Python CLI that gives the same trigger surface the Telegram bot exposes. The goal is end-to-end iteration on the bot's business logic without launching a real Telegram bot, without touching rex/siong databases, and without making automated calls to cm99.net unless the developer explicitly invokes one. + +## Non-Goals + +- Importing a sanitized snapshot of rex/siong data into the dev DB. Seed data only; snapshot import can come later. +- Containerizing the new bot CLI. It runs from the local Python virtualenv against a port-published dev mysql. +- Changing the prod compose file (`docker-compose.yml`). Portainer stacks for rex/siong remain untouched. +- Migrating or rotating the existing committed `envs/rex/.env` and `envs/siong/.env` files. Those are an R2 problem. +- Building or running `telegram-bot` or `transfer-bot` automatically in dev. They are gated behind a compose `bots` profile and stay quiet. +- Adding curses/textual/prompt_toolkit. The interactive TUI uses stdlib `input()` only. +- Adding a `remove`/`delete` CLI subcommand. The current Telegram bot has no analog and the schema has no soft-delete state. + +## Architecture Overview + +Three additions, no removals: + +1. **A `mysql` service in `docker-compose.override.yml`** — MySQL 8.0 image, named volume for persistence, init scripts mounted at `/docker-entrypoint-initdb.d/`. Published to `127.0.0.1:3306:3306` so the local CLI (running outside Docker) can reach it without exposing the port to anything else on the network. +2. **A new Python CLI module: `app/bot_cli.py`** — argparse-driven, mirrors the four Telegram bot handlers plus three operational ops. With no arguments, drops into a stdlib interactive menu (TUI-style; not full curses). +3. **Two shell scripts in `scripts/`** — `dev.sh` (lifecycle: `up`/`down`/`reset-db`/`logs`/`status`) and `bot_cli.sh` (env-loading wrapper for `python -m app.bot_cli`). + +Compose service start matrix: + +| Command | mysql | api-server | web-view | telegram-bot | transfer-bot | +|---|---|---|---|---|---| +| `dev.sh up` | ✓ | ✓ | ✓ | (gated by `bots` profile) | (gated by `bots` profile) | +| `bot_cli.sh ...` | (already up — E2 check) | (already up) | (already up) | — (CLI is local Python) | — | +| Prod (Portainer using `docker-compose.yml`) | — | ✓ | ✓ | ✓ | ✓ | + +Network: api-server reaches `mysql:3306` over the compose network; the local CLI reaches `127.0.0.1:3306` via the published port. `DB_HOST` is overridden by `bot_cli.sh` so a single `.env` works for both. + +## Components + +### 1. `docker-compose.override.yml` changes + +Add a `mysql` service to the existing override file (which currently only has `build:` directives for local image builds): + +```yaml +services: + mysql: + image: mysql:8.0 + container_name: ${CM_DEPLOY_NAME:-cm}-mysql + restart: unless-stopped + environment: + MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-devroot} + MYSQL_DATABASE: ${DB_NAME} + MYSQL_USER: ${DB_USER} + MYSQL_PASSWORD: ${DB_PASSWORD} + ports: + - "127.0.0.1:3306:3306" + volumes: + - mysql-data:/var/lib/mysql + - ./docker/mysql/init.d:/docker-entrypoint-initdb.d:ro + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + healthcheck: + test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-u", "root", "-p${MYSQL_ROOT_PASSWORD:-devroot}"] + interval: 5s + timeout: 3s + retries: 12 + networks: + - bot-network + + api-server: + depends_on: + mysql: + condition: service_healthy + + telegram-bot: + profiles: ["bots"] + + transfer-bot: + profiles: ["bots"] + +volumes: + mysql-data: + name: ${CM_DEPLOY_NAME:-cm}-mysql-data +``` + +Notes: +- The `127.0.0.1:` host-binding prefix means port 3306 is reachable from your shell only — not from other machines on the LAN. Important: the rex/siong scanner at `192.168.0.210` cannot reach this even if it tried. +- `depends_on: { mysql: { condition: service_healthy } }` means `api-server` waits for the healthcheck to pass before starting. This eliminates the cold-start race where api-server tried to connect before mysql was accepting traffic. +- Adding `profiles: ["bots"]` to `telegram-bot` and `transfer-bot` keeps them out of `docker compose up` in dev, but `docker compose run --rm telegram-bot ...` still works for parity tests. + +### 2. `docker/mysql/init.d/` + +Two SQL scripts run once on first volume creation (mysql:8 invokes everything in `/docker-entrypoint-initdb.d/` in alphabetical order): + +- `docker/mysql/init.d/01-schema.sql` — exactly the DDL from `AGENTS.md` (acc + user tables, utf8mb4). The `CREATE DATABASE` from AGENTS.md is dropped because mysql:8 image creates `${DB_NAME}` via env. The `USE rex_cm` line is replaced with `USE \`${DB_NAME}\``. +- `docker/mysql/init.d/02-seed.sql` — minimal seed: one row in `acc` matching `CM_PREFIX_PATTERN=13c` so `get_next_username` works, plus three filler rows. + +Seed contents: + +```sql +INSERT INTO acc (username, password, status, link) VALUES + ('13c1000', 'seedpass', '', ''), + ('13c1001', 'seedpass', '', ''), + ('13c1002', 'seedpass', '', ''), + ('13c1003', 'seedpass', '', ''); +``` + +These are dev-only seed values — never real credentials. The scripts only run on a fresh volume (mysql refuses to re-init an initialized volume). To re-seed, run `dev.sh reset-db`. + +### 3. `app/bot_cli.py` + +```python +import argparse +import os +import sys + +from .cm_bot_hal import CM_BOT_HAL + + +def _print_user(user: dict) -> None: + print(f"Username: {user['username']}") + print(f"Password: {user['password']}") + print(f"Link: {user['link']}") + + +def cmd_register(_args): + bot = CM_BOT_HAL() + _print_user(bot.get_user_api()) + + +def cmd_set_pin(args): + bot = CM_BOT_HAL() + if not bot.is_whatsapp_url(args.link): + print(f"ERROR: not a WhatsApp URL: {args.link}", file=sys.stderr) + sys.exit(2) + # Resolve names locally so we have something useful to print regardless of + # what set_security_pin_api returns. The HAL currently returns a bool from + # the trailing insert_user_to_table_user; the Telegram handler has the + # same shape and a latent bug accessing result['f_username']. We avoid + # depending on the return shape here. + t_username, f_username = bot.get_whatsapp_link_username(args.link) + success = bot.set_security_pin_api(args.link) + if not success: + print("ERROR: set_security_pin_api returned a falsy result", file=sys.stderr) + sys.exit(1) + print(f"OK: f_username={f_username} t_username={t_username}") + + +def cmd_insert_user(args): + bot = CM_BOT_HAL() + f_password = bot.get_user_pass_from_acc(args.f_username) + if not f_password: + print(f"ERROR: no password for {args.f_username}", file=sys.stderr) + sys.exit(2) + success = bot.insert_user_to_table_user({ + 'f_username': args.f_username, + 'f_password': f_password, + 't_username': args.t_username, + 't_password': bot.security_pin, + }) + if not success: + print("ERROR: insert failed", file=sys.stderr) + sys.exit(1) + print(f"OK: inserted {args.f_username} → {args.t_username}") + + +def cmd_credit(args): + bot = CM_BOT_HAL() + print(f"Credit: {bot.get_user_credit(args.username, args.password)}") + + +def cmd_transfer(args): + bot = CM_BOT_HAL() + print(bot.transfer_credit_api(args.f_username, args.f_password, args.t_username, args.t_password)) + + +def cmd_monitor_once(args): + bot = CM_BOT_HAL() + available = bot.get_all_available_acc() + print(f"Available accounts: {len(available)} (target: {args.target})") + if len(available) >= args.target: + print("Already at target; nothing to do.") + return + for _ in range(len(available), args.target): + try: + user = bot.create_new_acc() + print(f"Created: {user['username']}") + except Exception as exc: + print(f"ERROR creating account: {exc}", file=sys.stderr) + sys.exit(1) + + +def cmd_interactive(_args): + """Telegram-style menu in a TTY loop. stdlib only.""" + print("CM Bot CLI — interactive (type 'q' to quit, '?' for menu)") + while True: + print() + print(" 1 Register / get next account") + print(" 2 Set security PIN") + print(" 3 Insert into user table") + print(" credit Read account credit") + print(" transfer One-shot credit transfer") + print(" monitor [N] Run monitor once (default 20)") + print(" q Quit") + try: + line = input("> ").strip() + except (EOFError, KeyboardInterrupt): + print() + return + if not line: + continue + if line in ("q", "quit", "exit"): + return + if line in ("?", "help", "menu"): + continue + argv = line.split() + # Map TUI shortcuts to argparse subcommand names so we reuse the same + # dispatch table for both modes. + TUI_ALIASES = {"1": "register", "2": "set-pin", "3": "insert-user"} + argv[0] = TUI_ALIASES.get(argv[0], argv[0]) + try: + args = build_parser().parse_args(argv) + args.func(args) + except SystemExit: + # argparse calls sys.exit() on parse error; swallow it to keep the + # REPL alive instead of bailing out of the loop. + continue + except Exception as exc: + print(f"ERROR: {exc}", file=sys.stderr) + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(prog="bot_cli", description="CM Bot dev CLI (mirrors Telegram triggers).") + sub = p.add_subparsers(dest="command") + + sp = sub.add_parser("register", aliases=["get-acc"], help="Get next available account (Telegram /1).") + sp.set_defaults(func=cmd_register) + + sp = sub.add_parser("set-pin", help="Set security PIN from a WhatsApp link (Telegram /2).") + sp.add_argument("link") + sp.set_defaults(func=cmd_set_pin) + + sp = sub.add_parser("insert-user", help="Insert into user table (Telegram /3).") + sp.add_argument("f_username") + sp.add_argument("t_username") + sp.set_defaults(func=cmd_insert_user) + + sp = sub.add_parser("credit", help="Read account credit balance.") + sp.add_argument("username") + sp.add_argument("password") + sp.set_defaults(func=cmd_credit) + + sp = sub.add_parser("transfer", help="One-shot credit transfer.") + sp.add_argument("f_username") + sp.add_argument("f_password") + sp.add_argument("t_username") + sp.add_argument("t_password") + sp.set_defaults(func=cmd_transfer) + + sp = sub.add_parser("monitor-once", aliases=["monitor"], help="One iteration of the auto-create monitor.") + sp.add_argument("--target", type=int, default=20) + sp.set_defaults(func=cmd_monitor_once) + + sp = sub.add_parser("interactive", help="Drop into the TUI menu.") + sp.set_defaults(func=cmd_interactive) + + return p + + +def main(argv=None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + if args.command is None: + # Default mode: interactive TUI. + return cmd_interactive(args) or 0 + return args.func(args) or 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +Design notes: + +- The TUI loop reuses `argparse` parsing so subcommand and interactive paths share validation. SystemExit is caught so a typo (e.g., `2` with no link) prints the argparse error and returns to the prompt instead of killing the REPL. +- The default `python -m app.bot_cli` (no args) drops into interactive mode. This matches the user's "TUI" expectation while keeping the same module usable as a one-shot script. +- Subcommand names are spelled-out verbs (`register`, `set-pin`) for shell scripting; the TUI accepts `1`/`2`/`3` shortcuts so muscle memory from Telegram works. +- The CLI imports `CM_BOT_HAL` directly. No new business logic. If you change the bot's behavior in `cm_bot_hal.py`, the CLI tracks automatically. + +### 4. `scripts/dev.sh` + +```bash +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'EOF' +Lifecycle commands for the local dev stack (mysql + api-server + web-view). +Bots (telegram-bot, transfer-bot) are gated behind a compose 'bots' profile +and do not start with 'up'. + +Usage: + scripts/dev.sh up Start the dev stack in the background. + scripts/dev.sh down Stop the stack. Volume kept (DB persists). + scripts/dev.sh reset-db Stop the stack AND drop the mysql volume. + scripts/dev.sh logs Tail logs from the running stack. + scripts/dev.sh status Print 'OK' if mysql container is running, else exit 1. +EOF +} + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "${ROOT_DIR}" + +SUDO="sudo" +[[ "${NO_SUDO:-0}" == "1" ]] && SUDO="" +COMPOSE=(${SUDO} docker compose -f docker-compose.yml -f docker-compose.override.yml) + +[[ -f .env ]] || { echo "ERROR: .env not found. cp envs/dev/.env.example .env (then edit)." >&2; exit 2; } + +case "${1:-}" in + up) + "${COMPOSE[@]}" up -d --build mysql api-server web-view + "${COMPOSE[@]}" ps + ;; + down) + "${COMPOSE[@]}" down + ;; + reset-db) + "${COMPOSE[@]}" down --volumes + "${COMPOSE[@]}" up -d --build mysql api-server web-view + ;; + logs) + "${COMPOSE[@]}" logs -f mysql api-server web-view + ;; + status) + if "${COMPOSE[@]}" ps --status running --services | grep -q '^mysql$'; then + echo OK + else + echo "ERROR: dev stack not running. Run 'scripts/dev.sh up' first." >&2 + exit 1 + fi + ;; + -h|--help|help|"") + usage + [[ "${1:-}" == "" ]] && exit 1 || exit 0 + ;; + *) + echo "unknown command: $1" >&2 + usage >&2 + exit 1 + ;; +esac +``` + +### 5. `scripts/bot_cli.sh` + +```bash +#!/usr/bin/env bash +# Run the bot CLI in the local venv. With no args, drops into the TUI menu. +# Requires: dev stack up (run scripts/dev.sh up first), .venv with deps. +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "${ROOT_DIR}" + +# E2: bail if the dev stack is not running. +if ! NO_SUDO="${NO_SUDO:-0}" bash scripts/dev.sh status >/dev/null 2>&1; then + echo "ERROR: dev stack not running. Run 'scripts/dev.sh up' first." >&2 + exit 2 +fi + +[[ -f .env ]] || { echo "ERROR: .env not found. cp envs/dev/.env.example .env (then edit)." >&2; exit 2; } + +# Load .env into the environment (export everything between 'set -a' and 'set +a'). +set -a +# shellcheck disable=SC1091 +source .env +set +a + +# Override DB host/port for the local CLI: docker mysql is published on +# 127.0.0.1:3306 even though api-server in-network reaches it as mysql:3306. +export DB_HOST=127.0.0.1 +export DB_PORT=3306 + +PYTHON_BIN="${PYTHON_BIN:-${ROOT_DIR}/.venv/bin/python}" +[[ -x "${PYTHON_BIN}" ]] || { echo "ERROR: ${PYTHON_BIN} not found. Create venv: python3 -m venv .venv && .venv/bin/pip install -r requirements.txt" >&2; exit 2; } + +exec "${PYTHON_BIN}" -m app.bot_cli "$@" +``` + +### 6. `envs/dev/.env.example` (committed) + +``` +# === Runtime === +CM_DEBUG=true + +# === Deployment Identity === +CM_DEPLOY_NAME=dev-cm +CM_WEB_HOST_PORT=8000 + +# === Docker Registry / Build === +CM_IMAGE_PREFIX=local +DOCKER_IMAGE_TAG=dev + +# === Telegram (unused in A2 — telegram-bot is gated by 'bots' profile) === +TELEGRAM_BOT_TOKEN=fill-only-if-running-bots-profile +TELEGRAM_ALERT_CHAT_ID= +TELEGRAM_ALERT_BOT_TOKEN= + +# === Database (dev mysql in docker; bot_cli.sh overrides DB_HOST=127.0.0.1) === +DB_HOST=mysql +DB_USER=cm +DB_PASSWORD=devpassword +DB_NAME=cm +DB_PORT=3306 +DB_CONNECTION_TIMEOUT=8 +DB_CONNECT_RETRIES=5 +DB_CONNECT_RETRY_DELAY=2 +MYSQL_ROOT_PASSWORD=devroot + +# === Bot Config === +# CM_PREFIX_PATTERN=13c MUST match the seed in docker/mysql/init.d/02-seed.sql. +CM_PREFIX_PATTERN=13c +CM_AGENT_ID=fill-with-real-agent-id-to-test-cm99-calls +CM_AGENT_PASSWORD=fill-with-real-agent-password-to-test-cm99-calls +CM_SECURITY_PIN=000000 +CM_BOT_BASE_URL=https://cm99.net +``` + +Operator workflow: `cp envs/dev/.env.example .env`, fill in real `CM_AGENT_ID` / `CM_AGENT_PASSWORD` only if they want to invoke `register` / `monitor-once` / `credit` / `transfer` against real cm99.net. The `set-pin` and `insert-user` ops also need them. + +`envs/dev/.env` (the operator's filled-in copy) is added to `.gitignore`. + +### 7. `.gitignore` change + +``` +__pycache__ +.DS_Store +*.html +logs +envs/dev/.env +``` + +Just `envs/dev/.env` — rex/siong stay tracked (their handling is R2's problem). + +### 8. `AGENTS.md` updates + +- Replace the section recommending raw schema setup with a pointer to `bash scripts/dev.sh up`. +- Add a new "Dev Tier" subsection: "Local development uses `envs/dev/.env.example` → `.env` → `bash scripts/dev.sh up`. The bot CLI: `bash scripts/bot_cli.sh` (TUI) or `bash scripts/bot_cli.sh `." +- Note: the auto-create monitor does NOT run in dev (it lives in `telegram-bot` which is gated by the `bots` profile). Use `bot_cli.sh monitor-once` to exercise that code path manually. + +## Files Created / Modified + +| File | Operation | +|---|---| +| `docker-compose.override.yml` | Modify — add mysql service, profile-gate bots, depends_on, volume | +| `docker/mysql/init.d/01-schema.sql` | Create | +| `docker/mysql/init.d/02-seed.sql` | Create | +| `app/bot_cli.py` | Create | +| `scripts/dev.sh` | Create | +| `scripts/bot_cli.sh` | Create | +| `envs/dev/.env.example` | Create | +| `.gitignore` | Modify — add `envs/dev/.env` | +| `AGENTS.md` | Modify — dev tier docs | + +No new Python or system dependencies. The bot CLI uses argparse + stdlib only; mysql:8.0 is a stable upstream image; everything else is shell + YAML. + +## Verification + +1. **Cold start.** `cp envs/dev/.env.example .env` → `bash scripts/dev.sh up`. After the healthcheck settles (~10–20s), `docker compose ps` shows mysql, api-server, web-view all `running`. `bash scripts/dev.sh status` prints `OK`. +2. **Schema + seed.** `mysql -h 127.0.0.1 -u cm -pdevpassword cm -e "SELECT username FROM acc"` returns the four seed rows. +3. **API smoke.** `curl http://localhost:3000/acc/` returns the four rows as JSON. `curl http://localhost:8000/api/acc/` proxies the same through the web-view. +4. **CLI no-args = TUI.** `bash scripts/bot_cli.sh` drops into the menu. Pressing `q` exits cleanly. +5. **CLI subcommand parity.** `bash scripts/bot_cli.sh register` (with real agent creds) returns a username/password/link triple matching what Telegram `/1` would return. `bash scripts/bot_cli.sh monitor-once --target 5` reports current pool size and creates accounts up to the target. +6. **Strict mode (E2).** `bash scripts/dev.sh down`, then `bash scripts/bot_cli.sh register` exits 2 with `ERROR: dev stack not running.` +7. **Reset.** `bash scripts/dev.sh reset-db` wipes the volume; on next `up`, the seed is re-applied (verified by step 2). +8. **Prod compose untouched.** `docker compose -f docker-compose.yml config` renders no mysql service, no bots profile gate, no extra ports — identical output to before this change. (rex/siong Portainer stacks unaffected.) + +## Risk + +Low. Three areas worth naming: + +- **Port collision.** `127.0.0.1:3306` collides with a host-side mysql server if one is already running locally. Operators with such a setup would change the host port (`MYSQL_HOST_PORT` env override is *not* added in this design — YAGNI; the .env can map it manually if needed). +- **bot_cli accidentally hitting prod cm99.net.** `register`, `set-pin`, `monitor-once`, `credit`, `transfer` all call cm99.net with real agent credentials. There is no "dry-run" flag in this design. The mitigation is documentation in `envs/dev/.env.example` (placeholder agent creds; operator opts in by replacing them) plus the AGENTS.md note. If a sandbox cm99 environment is ever available, swap `CM_BOT_BASE_URL`. +- **mysql startup race after `reset-db`.** The healthcheck handles this — api-server waits — so it's covered, but worth listing. +- **Latent bug in `set_security_pin_api`'s return contract.** The HAL returns a bool but `cm_telegram.py:87` does `result['f_username']` on it, which would TypeError in the success branch (currently swallowed by the surrounding except). The CLI's `cmd_set_pin` works around it by re-resolving names locally. Fixing the contract belongs in a separate change (HAL refactor), out of scope here so we don't expand A's blast radius. + +## Out-of-Scope Follow-Ups + +- **Sanitized prod snapshot import** — `dev.sh import-snapshot rex` that mysqldumps prod and scrubs sensitive columns before loading. Useful but separate scope. +- **`--dry-run` flag for bot_cli** — record-only mode for `register`/`transfer`. Useful for testing the CLI plumbing without touching cm99.net. +- **Local `web-view` hot-reload** — currently the dev stack rebuilds the web image when files change; a bind mount + flask `--reload` (gated on `CM_DEBUG=true`) would make UI iteration instant. R3-adjacent but mostly an A++ improvement. +- **Migrate rex/siong off committed env files** — the R2 effort. Independent of A.