yiekheng/cm_bot_v2
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add design spec for local-as-dev tier (sub-project A)

Browse Source 
Adds containerized MySQL to docker-compose.override.yml, gates
telegram/transfer bots behind a 'bots' profile, and introduces a local
Python bot CLI with a stdlib TUI menu that mirrors Telegram's /1, /2,
/3 plus operational subcommands. CLI runs from .venv against
127.0.0.1:3306 (mysql published to localhost only).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
yiekheng 2026-05-02 16:49:12 +08:00
parent 9db3980304
commit 94ef5595ea
1 changed files with 499 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

499
docs/superpowers/specs/2026-05-02-local-as-dev-design.md Normal file
View File

@ -0,0 +1,499 @@
# Local-as-Dev Tier (Sub-Project A) Design
**Date:** 2026-05-02
**Status:** Approved (design)
**Sequel to:** [2026-05-02-debug-mode-hotfix-design.md](2026-05-02-debug-mode-hotfix-design.md)
**Out of scope (separate):** rex/siong env file rotation (R2), cm_bot.py scraper resilience (R3), security hardening sub-project (C), Next.js web view (B).
## Problem
`cm_bot_v2` runs two production deployments — `rex` (port 8001) and `siong` (port 8005) — both deployed via Portainer from images on `gitea.04080616.xyz/yiekheng`. There is no formal local development tier. Local iteration today either touches a real production database or has no DB at all (`web-view` is the only service that boots without one). This blocks safe testing of the security hardening (sub-project C) and the planned Next.js rewrite (B), both of which need to be exercised end-to-end without poking prod data.
## Goal
Add a local-only "dev tier" that spins up `mysql` + `api-server` + `web-view` self-contained on a developer machine, plus a Python CLI that gives the same trigger surface the Telegram bot exposes. The goal is end-to-end iteration on the bot's business logic without launching a real Telegram bot, without touching rex/siong databases, and without making automated calls to cm99.net unless the developer explicitly invokes one.
## Non-Goals
- Importing a sanitized snapshot of rex/siong data into the dev DB. Seed data only; snapshot import can come later.
- Containerizing the new bot CLI. It runs from the local Python virtualenv against a port-published dev mysql.
- Changing the prod compose file (`docker-compose.yml`). Portainer stacks for rex/siong remain untouched.
- Migrating or rotating the existing committed `envs/rex/.env` and `envs/siong/.env` files. Those are an R2 problem.
- Building or running `telegram-bot` or `transfer-bot` automatically in dev. They are gated behind a compose `bots` profile and stay quiet.
- Adding curses/textual/prompt_toolkit. The interactive TUI uses stdlib `input()` only.
- Adding a `remove`/`delete` CLI subcommand. The current Telegram bot has no analog and the schema has no soft-delete state.
## Architecture Overview
Three additions, no removals:
1. **A `mysql` service in `docker-compose.override.yml`** — MySQL 8.0 image, named volume for persistence, init scripts mounted at `/docker-entrypoint-initdb.d/`. Published to `127.0.0.1:3306:3306` so the local CLI (running outside Docker) can reach it without exposing the port to anything else on the network.
2. **A new Python CLI module: `app/bot_cli.py`** — argparse-driven, mirrors the four Telegram bot handlers plus three operational ops. With no arguments, drops into a stdlib interactive menu (TUI-style; not full curses).
3. **Two shell scripts in `scripts/`** — `dev.sh` (lifecycle: `up`/`down`/`reset-db`/`logs`/`status`) and `bot_cli.sh` (env-loading wrapper for `python -m app.bot_cli`).
Compose service start matrix:
| Command | mysql | api-server | web-view | telegram-bot | transfer-bot |
|---|---|---|---|---|---|
| `dev.sh up` | ✓ | ✓ | ✓ | (gated by `bots` profile) | (gated by `bots` profile) |
| `bot_cli.sh ...` | (already up — E2 check) | (already up) | (already up) | — (CLI is local Python) | — |
| Prod (Portainer using `docker-compose.yml`) | — | ✓ | ✓ | ✓ | ✓ |
Network: api-server reaches `mysql:3306` over the compose network; the local CLI reaches `127.0.0.1:3306` via the published port. `DB_HOST` is overridden by `bot_cli.sh` so a single `.env` works for both.
## Components
### 1. `docker-compose.override.yml` changes
Add a `mysql` service to the existing override file (which currently only has `build:` directives for local image builds):
```yaml
services:
mysql:
image: mysql:8.0
container_name: ${CM_DEPLOY_NAME:-cm}-mysql
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-devroot}
MYSQL_DATABASE: ${DB_NAME}
MYSQL_USER: ${DB_USER}
MYSQL_PASSWORD: ${DB_PASSWORD}
ports:
- "127.0.0.1:3306:3306"
volumes:
- mysql-data:/var/lib/mysql
- ./docker/mysql/init.d:/docker-entrypoint-initdb.d:ro
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-u", "root", "-p${MYSQL_ROOT_PASSWORD:-devroot}"]
interval: 5s
timeout: 3s
retries: 12
networks:
- bot-network
api-server:
depends_on:
mysql:
condition: service_healthy
telegram-bot:
profiles: ["bots"]
transfer-bot:
profiles: ["bots"]
volumes:
mysql-data:
name: ${CM_DEPLOY_NAME:-cm}-mysql-data
```
Notes:
- The `127.0.0.1:` host-binding prefix means port 3306 is reachable from your shell only — not from other machines on the LAN. Important: the rex/siong scanner at `192.168.0.210` cannot reach this even if it tried.
- `depends_on: { mysql: { condition: service_healthy } }` means `api-server` waits for the healthcheck to pass before starting. This eliminates the cold-start race where api-server tried to connect before mysql was accepting traffic.
- Adding `profiles: ["bots"]` to `telegram-bot` and `transfer-bot` keeps them out of `docker compose up` in dev, but `docker compose run --rm telegram-bot ...` still works for parity tests.
### 2. `docker/mysql/init.d/`
Two SQL scripts run once on first volume creation (mysql:8 invokes everything in `/docker-entrypoint-initdb.d/` in alphabetical order):
- `docker/mysql/init.d/01-schema.sql` — exactly the DDL from `AGENTS.md` (acc + user tables, utf8mb4). The `CREATE DATABASE` from AGENTS.md is dropped because mysql:8 image creates `${DB_NAME}` via env. The `USE rex_cm` line is replaced with `USE \`${DB_NAME}\``.
- `docker/mysql/init.d/02-seed.sql` — minimal seed: one row in `acc` matching `CM_PREFIX_PATTERN=13c` so `get_next_username` works, plus three filler rows.
Seed contents:
```sql
INSERT INTO acc (username, password, status, link) VALUES
('13c1000', 'seedpass', '', ''),
('13c1001', 'seedpass', '', ''),
('13c1002', 'seedpass', '', ''),
('13c1003', 'seedpass', '', '');
```
These are dev-only seed values — never real credentials. The scripts only run on a fresh volume (mysql refuses to re-init an initialized volume). To re-seed, run `dev.sh reset-db`.
### 3. `app/bot_cli.py`
```python
import argparse
import os
import sys
from .cm_bot_hal import CM_BOT_HAL
def _print_user(user: dict) -> None:
print(f"Username: {user['username']}")
print(f"Password: {user['password']}")
print(f"Link: {user['link']}")
def cmd_register(_args):
bot = CM_BOT_HAL()
_print_user(bot.get_user_api())
def cmd_set_pin(args):
bot = CM_BOT_HAL()
if not bot.is_whatsapp_url(args.link):
print(f"ERROR: not a WhatsApp URL: {args.link}", file=sys.stderr)
sys.exit(2)
# Resolve names locally so we have something useful to print regardless of
# what set_security_pin_api returns. The HAL currently returns a bool from
# the trailing insert_user_to_table_user; the Telegram handler has the
# same shape and a latent bug accessing result['f_username']. We avoid
# depending on the return shape here.
t_username, f_username = bot.get_whatsapp_link_username(args.link)
success = bot.set_security_pin_api(args.link)
if not success:
print("ERROR: set_security_pin_api returned a falsy result", file=sys.stderr)
sys.exit(1)
print(f"OK: f_username={f_username} t_username={t_username}")
def cmd_insert_user(args):
bot = CM_BOT_HAL()
f_password = bot.get_user_pass_from_acc(args.f_username)
if not f_password:
print(f"ERROR: no password for {args.f_username}", file=sys.stderr)
sys.exit(2)
success = bot.insert_user_to_table_user({
'f_username': args.f_username,
'f_password': f_password,
't_username': args.t_username,
't_password': bot.security_pin,
})
if not success:
print("ERROR: insert failed", file=sys.stderr)
sys.exit(1)
print(f"OK: inserted {args.f_username} → {args.t_username}")
def cmd_credit(args):
bot = CM_BOT_HAL()
print(f"Credit: {bot.get_user_credit(args.username, args.password)}")
def cmd_transfer(args):
bot = CM_BOT_HAL()
print(bot.transfer_credit_api(args.f_username, args.f_password, args.t_username, args.t_password))
def cmd_monitor_once(args):
bot = CM_BOT_HAL()
available = bot.get_all_available_acc()
print(f"Available accounts: {len(available)} (target: {args.target})")
if len(available) >= args.target:
print("Already at target; nothing to do.")
return
for _ in range(len(available), args.target):
try:
user = bot.create_new_acc()
print(f"Created: {user['username']}")
except Exception as exc:
print(f"ERROR creating account: {exc}", file=sys.stderr)
sys.exit(1)
def cmd_interactive(_args):
"""Telegram-style menu in a TTY loop. stdlib only."""
print("CM Bot CLI — interactive (type 'q' to quit, '?' for menu)")
while True:
print()
print(" 1 Register / get next account")
print(" 2 <whatsapp_link> Set security PIN")
print(" 3 <f_username> <t_username> Insert into user table")
print(" credit <username> <password> Read account credit")
print(" transfer <fu> <fp> <tu> <tp> One-shot credit transfer")
print(" monitor [N] Run monitor once (default 20)")
print(" q Quit")
try:
line = input("> ").strip()
except (EOFError, KeyboardInterrupt):
print()
return
if not line:
continue
if line in ("q", "quit", "exit"):
return
if line in ("?", "help", "menu"):
continue
argv = line.split()
# Map TUI shortcuts to argparse subcommand names so we reuse the same
# dispatch table for both modes.
TUI_ALIASES = {"1": "register", "2": "set-pin", "3": "insert-user"}
argv[0] = TUI_ALIASES.get(argv[0], argv[0])
try:
args = build_parser().parse_args(argv)
args.func(args)
except SystemExit:
# argparse calls sys.exit() on parse error; swallow it to keep the
# REPL alive instead of bailing out of the loop.
continue
except Exception as exc:
print(f"ERROR: {exc}", file=sys.stderr)
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="bot_cli", description="CM Bot dev CLI (mirrors Telegram triggers).")
sub = p.add_subparsers(dest="command")
sp = sub.add_parser("register", aliases=["get-acc"], help="Get next available account (Telegram /1).")
sp.set_defaults(func=cmd_register)
sp = sub.add_parser("set-pin", help="Set security PIN from a WhatsApp link (Telegram /2).")
sp.add_argument("link")
sp.set_defaults(func=cmd_set_pin)
sp = sub.add_parser("insert-user", help="Insert into user table (Telegram /3).")
sp.add_argument("f_username")
sp.add_argument("t_username")
sp.set_defaults(func=cmd_insert_user)
sp = sub.add_parser("credit", help="Read account credit balance.")
sp.add_argument("username")
sp.add_argument("password")
sp.set_defaults(func=cmd_credit)
sp = sub.add_parser("transfer", help="One-shot credit transfer.")
sp.add_argument("f_username")
sp.add_argument("f_password")
sp.add_argument("t_username")
sp.add_argument("t_password")
sp.set_defaults(func=cmd_transfer)
sp = sub.add_parser("monitor-once", aliases=["monitor"], help="One iteration of the auto-create monitor.")
sp.add_argument("--target", type=int, default=20)
sp.set_defaults(func=cmd_monitor_once)
sp = sub.add_parser("interactive", help="Drop into the TUI menu.")
sp.set_defaults(func=cmd_interactive)
return p
def main(argv=None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
if args.command is None:
# Default mode: interactive TUI.
return cmd_interactive(args) or 0
return args.func(args) or 0
if __name__ == "__main__":
sys.exit(main())
```
Design notes:
- The TUI loop reuses `argparse` parsing so subcommand and interactive paths share validation. SystemExit is caught so a typo (e.g., `2` with no link) prints the argparse error and returns to the prompt instead of killing the REPL.
- The default `python -m app.bot_cli` (no args) drops into interactive mode. This matches the user's "TUI" expectation while keeping the same module usable as a one-shot script.
- Subcommand names are spelled-out verbs (`register`, `set-pin`) for shell scripting; the TUI accepts `1`/`2`/`3` shortcuts so muscle memory from Telegram works.
- The CLI imports `CM_BOT_HAL` directly. No new business logic. If you change the bot's behavior in `cm_bot_hal.py`, the CLI tracks automatically.
### 4. `scripts/dev.sh`
```bash
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'EOF'
Lifecycle commands for the local dev stack (mysql + api-server + web-view).
Bots (telegram-bot, transfer-bot) are gated behind a compose 'bots' profile
and do not start with 'up'.
Usage:
scripts/dev.sh up Start the dev stack in the background.
scripts/dev.sh down Stop the stack. Volume kept (DB persists).
scripts/dev.sh reset-db Stop the stack AND drop the mysql volume.
scripts/dev.sh logs Tail logs from the running stack.
scripts/dev.sh status Print 'OK' if mysql container is running, else exit 1.
EOF
}
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${ROOT_DIR}"
SUDO="sudo"
[[ "${NO_SUDO:-0}" == "1" ]] && SUDO=""
COMPOSE=(${SUDO} docker compose -f docker-compose.yml -f docker-compose.override.yml)
[[ -f .env ]] || { echo "ERROR: .env not found. cp envs/dev/.env.example .env (then edit)." >&2; exit 2; }
case "${1:-}" in
up)
"${COMPOSE[@]}" up -d --build mysql api-server web-view
"${COMPOSE[@]}" ps
;;
down)
"${COMPOSE[@]}" down
;;
reset-db)
"${COMPOSE[@]}" down --volumes
"${COMPOSE[@]}" up -d --build mysql api-server web-view
;;
logs)
"${COMPOSE[@]}" logs -f mysql api-server web-view
;;
status)
if "${COMPOSE[@]}" ps --status running --services | grep -q '^mysql$'; then
echo OK
else
echo "ERROR: dev stack not running. Run 'scripts/dev.sh up' first." >&2
exit 1
fi
;;
-h|--help|help|"")
usage
[[ "${1:-}" == "" ]] && exit 1 || exit 0
;;
*)
echo "unknown command: $1" >&2
usage >&2
exit 1
;;
esac
```
### 5. `scripts/bot_cli.sh`
```bash
#!/usr/bin/env bash
# Run the bot CLI in the local venv. With no args, drops into the TUI menu.
# Requires: dev stack up (run scripts/dev.sh up first), .venv with deps.
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${ROOT_DIR}"
# E2: bail if the dev stack is not running.
if ! NO_SUDO="${NO_SUDO:-0}" bash scripts/dev.sh status >/dev/null 2>&1; then
echo "ERROR: dev stack not running. Run 'scripts/dev.sh up' first." >&2
exit 2
fi
[[ -f .env ]] || { echo "ERROR: .env not found. cp envs/dev/.env.example .env (then edit)." >&2; exit 2; }
# Load .env into the environment (export everything between 'set -a' and 'set +a').
set -a
# shellcheck disable=SC1091
source .env
set +a
# Override DB host/port for the local CLI: docker mysql is published on
# 127.0.0.1:3306 even though api-server in-network reaches it as mysql:3306.
export DB_HOST=127.0.0.1
export DB_PORT=3306
PYTHON_BIN="${PYTHON_BIN:-${ROOT_DIR}/.venv/bin/python}"
[[ -x "${PYTHON_BIN}" ]] || { echo "ERROR: ${PYTHON_BIN} not found. Create venv: python3 -m venv .venv && .venv/bin/pip install -r requirements.txt" >&2; exit 2; }
exec "${PYTHON_BIN}" -m app.bot_cli "$@"
```
### 6. `envs/dev/.env.example` (committed)
```
# === Runtime ===
CM_DEBUG=true
# === Deployment Identity ===
CM_DEPLOY_NAME=dev-cm
CM_WEB_HOST_PORT=8000
# === Docker Registry / Build ===
CM_IMAGE_PREFIX=local
DOCKER_IMAGE_TAG=dev
# === Telegram (unused in A2 — telegram-bot is gated by 'bots' profile) ===
TELEGRAM_BOT_TOKEN=fill-only-if-running-bots-profile
TELEGRAM_ALERT_CHAT_ID=
TELEGRAM_ALERT_BOT_TOKEN=
# === Database (dev mysql in docker; bot_cli.sh overrides DB_HOST=127.0.0.1) ===
DB_HOST=mysql
DB_USER=cm
DB_PASSWORD=devpassword
DB_NAME=cm
DB_PORT=3306
DB_CONNECTION_TIMEOUT=8
DB_CONNECT_RETRIES=5
DB_CONNECT_RETRY_DELAY=2
MYSQL_ROOT_PASSWORD=devroot
# === Bot Config ===
# CM_PREFIX_PATTERN=13c MUST match the seed in docker/mysql/init.d/02-seed.sql.
CM_PREFIX_PATTERN=13c
CM_AGENT_ID=fill-with-real-agent-id-to-test-cm99-calls
CM_AGENT_PASSWORD=fill-with-real-agent-password-to-test-cm99-calls
CM_SECURITY_PIN=000000
CM_BOT_BASE_URL=https://cm99.net
```
Operator workflow: `cp envs/dev/.env.example .env`, fill in real `CM_AGENT_ID` / `CM_AGENT_PASSWORD` only if they want to invoke `register` / `monitor-once` / `credit` / `transfer` against real cm99.net. The `set-pin` and `insert-user` ops also need them.
`envs/dev/.env` (the operator's filled-in copy) is added to `.gitignore`.
### 7. `.gitignore` change
```
__pycache__
.DS_Store
*.html
logs
envs/dev/.env
```
Just `envs/dev/.env` — rex/siong stay tracked (their handling is R2's problem).
### 8. `AGENTS.md` updates
- Replace the section recommending raw schema setup with a pointer to `bash scripts/dev.sh up`.
- Add a new "Dev Tier" subsection: "Local development uses `envs/dev/.env.example` → `.env` → `bash scripts/dev.sh up`. The bot CLI: `bash scripts/bot_cli.sh` (TUI) or `bash scripts/bot_cli.sh <subcommand>`."
- Note: the auto-create monitor does NOT run in dev (it lives in `telegram-bot` which is gated by the `bots` profile). Use `bot_cli.sh monitor-once` to exercise that code path manually.
## Files Created / Modified
| File | Operation |
|---|---|
| `docker-compose.override.yml` | Modify — add mysql service, profile-gate bots, depends_on, volume |
| `docker/mysql/init.d/01-schema.sql` | Create |
| `docker/mysql/init.d/02-seed.sql` | Create |
| `app/bot_cli.py` | Create |
| `scripts/dev.sh` | Create |
| `scripts/bot_cli.sh` | Create |
| `envs/dev/.env.example` | Create |
| `.gitignore` | Modify — add `envs/dev/.env` |
| `AGENTS.md` | Modify — dev tier docs |
No new Python or system dependencies. The bot CLI uses argparse + stdlib only; mysql:8.0 is a stable upstream image; everything else is shell + YAML.
## Verification
1. **Cold start.** `cp envs/dev/.env.example .env` → `bash scripts/dev.sh up`. After the healthcheck settles (~10–20s), `docker compose ps` shows mysql, api-server, web-view all `running`. `bash scripts/dev.sh status` prints `OK`.
2. **Schema + seed.** `mysql -h 127.0.0.1 -u cm -pdevpassword cm -e "SELECT username FROM acc"` returns the four seed rows.
3. **API smoke.** `curl http://localhost:3000/acc/` returns the four rows as JSON. `curl http://localhost:8000/api/acc/` proxies the same through the web-view.
4. **CLI no-args = TUI.** `bash scripts/bot_cli.sh` drops into the menu. Pressing `q` exits cleanly.
5. **CLI subcommand parity.** `bash scripts/bot_cli.sh register` (with real agent creds) returns a username/password/link triple matching what Telegram `/1` would return. `bash scripts/bot_cli.sh monitor-once --target 5` reports current pool size and creates accounts up to the target.
6. **Strict mode (E2).** `bash scripts/dev.sh down`, then `bash scripts/bot_cli.sh register` exits 2 with `ERROR: dev stack not running.`
7. **Reset.** `bash scripts/dev.sh reset-db` wipes the volume; on next `up`, the seed is re-applied (verified by step 2).
8. **Prod compose untouched.** `docker compose -f docker-compose.yml config` renders no mysql service, no bots profile gate, no extra ports — identical output to before this change. (rex/siong Portainer stacks unaffected.)
## Risk
Low. Three areas worth naming:
- **Port collision.** `127.0.0.1:3306` collides with a host-side mysql server if one is already running locally. Operators with such a setup would change the host port (`MYSQL_HOST_PORT` env override is *not* added in this design — YAGNI; the .env can map it manually if needed).
- **bot_cli accidentally hitting prod cm99.net.** `register`, `set-pin`, `monitor-once`, `credit`, `transfer` all call cm99.net with real agent credentials. There is no "dry-run" flag in this design. The mitigation is documentation in `envs/dev/.env.example` (placeholder agent creds; operator opts in by replacing them) plus the AGENTS.md note. If a sandbox cm99 environment is ever available, swap `CM_BOT_BASE_URL`.
- **mysql startup race after `reset-db`.** The healthcheck handles this — api-server waits — so it's covered, but worth listing.
- **Latent bug in `set_security_pin_api`'s return contract.** The HAL returns a bool but `cm_telegram.py:87` does `result['f_username']` on it, which would TypeError in the success branch (currently swallowed by the surrounding except). The CLI's `cmd_set_pin` works around it by re-resolving names locally. Fixing the contract belongs in a separate change (HAL refactor), out of scope here so we don't expand A's blast radius.
## Out-of-Scope Follow-Ups
- **Sanitized prod snapshot import** — `dev.sh import-snapshot rex` that mysqldumps prod and scrubs sensitive columns before loading. Useful but separate scope.
- **`--dry-run` flag for bot_cli** — record-only mode for `register`/`transfer`. Useful for testing the CLI plumbing without touching cm99.net.
- **Local `web-view` hot-reload** — currently the dev stack rebuilds the web image when files change; a bind mount + flask `--reload` (gated on `CM_DEBUG=true`) would make UI iteration instant. R3-adjacent but mostly an A++ improvement.
- **Migrate rex/siong off committed env files** — the R2 effort. Independent of A.