cm_bot_v2/docs/superpowers/specs/2026-05-02-r3-scraper-resilience-design.md

R3: cm_bot.py Scraper Resilience Design

Date: 2026-05-02 Status: Approved (design) Scope: Tiny one-shot. Self-contained inside app/cm_bot.py plus tests. Out of scope: Caching Struts CSRF tokens — see "Why no token caching" below. Replacing the scraper with an HTTP-API client (cm99.net doesn't expose one — see Playwright probe earlier in this session).

Problem

app/cm_bot.py integrates with cm99.net by parsing HTML responses with BeautifulSoup. When the site re-skins or returns an unexpected page (login expired, session timeout, server error), the existing code fails with cryptic errors like TypeError: 'NoneType' object is not subscriptable because lines such as soup.find('input', {'name': 'token'})['value'] index None. The actual response body that caused the failure is lost — there is commented-out HTML-dump code (with open('credit-{now}.html', 'wb') as f: f.write(response.content)) hinting that the operator has done this manually before, but it is not active.

The user's broader question earlier in this session was whether to migrate cm99.net integration to a real API; we confirmed it can't because cm99.net is a 2017-era Java/Struts2 monolith that returns server-rendered HTML for everything. So we make the scraper robust instead.

Goal

Two narrow improvements:

1. Extract-or-raise helper. A _find_input_value(soup, name, context) instance method that replaces the bare soup.find('input', {'name': name})['value'] pattern. On miss, it dumps the failing response to logs/scraper-failures/<context>-<timestamp>.html and raises a ScraperError with the context and what it was looking for. Existing exception handlers in cm_bot_hal.py continue to catch and report.
2. HTML dump on parse failure. A _dump_html(context, content) instance method that writes the raw response bytes to logs/scraper-failures/. Called from the helper above, plus from one already-attempted-but-commented site (get_user_credit's text-parsing block) where the failure mode is more involved than a missing input.

Non-Goals

• Caching the Struts CSRF token across operations. The tokens are anti-CSRF tokens that the Struts2 framework typically regenerates per form-load. Caching one and re-using it for a second POST risks "Invalid token" errors that are silent unless something checks the response body. Without confirmed evidence that this Struts deployment treats tokens as session-scoped (the user has not characterized this), caching is a correctness risk for a perf win we don't measurably need. Skipped intentionally.
• Replacing the bot's requests-based transport. Already validated as the right shape (no API exists).
• Touching cm_telegram.py:87's result['f_username'] line — that was already fixed in commit 231ae69 (the HAL now returns a dict).
• Adding a retry loop on transient HTTP errors. The HAL has its own retry logic (DB) and the bot's exception model is "raise → telegram bot or transfer worker reports". Adding scraper retries would change behavior in ways the operator hasn't asked for.

Architecture

ScraperError exception class

A small subclass of Exception so callers can distinguish scraper-side problems from network errors. Lives at the top of app/cm_bot.py.

class ScraperError(Exception):
    """A cm99.net response did not contain the field we expected.

    The raw response is saved to logs/scraper-failures/ before this is
    raised; the message identifies which method failed and what was
    being looked for.
    """

_dump_html instance method

Writes the response content to logs/scraper-failures/<context>-<YYYYMMDD-HHMMSS>.html. Creates the directory if it doesn't exist. Prints a single line to stdout so the operator notices in the container logs.

def _dump_html(self, context: str, content: bytes) -> str:
    ts = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    out_dir = os.path.join("logs", "scraper-failures")
    os.makedirs(out_dir, exist_ok=True)
    path = os.path.join(out_dir, f"{context}-{ts}.html")
    with open(path, "wb") as f:
        f.write(content if isinstance(content, (bytes, bytearray)) else content.encode("utf-8", "replace"))
    print(f"[scraper-failure] dumped {context} response to {path}")
    return path

The return value is the path so the helper above can include it in the exception message.

_find_input_value instance method

The dominant pattern in cm_bot.py is <input name="X" value="..."> extraction. This helper covers it.

def _find_input_value(self, soup, name: str, *, context: str, raw: bytes) -> str:
    el = soup.find("input", {"name": name})
    if el is None or "value" not in el.attrs:
        path = self._dump_html(context, raw)
        raise ScraperError(
            f"{context}: input[name={name!r}] missing or has no value attribute "
            f"(response saved to {path})"
        )
    return el["value"]

The caller passes raw=response.content so the dumped file is the actual bytes received, not the BeautifulSoup-stringified version.

Call sites that adopt the helper

Five existing methods in app/cm_bot.py use the bare soup.find('input', {'name': 'token'})['value'] pattern. Each gets converted in one mechanical edit:

Method	Before	After
get_register_form_token (line ~344)	soup.find('input', {'name': 'token'})['value']	self._find_input_value(soup, 'token', context='register_form_token', raw=response.content)
get_security_pin_form_token (line ~357)	same	self._find_input_value(soup, 'token', context='security_pin_form_token', raw=response.content)
get_transfer_token (line ~463)	same	self._find_input_value(soup, 'token', context='transfer_token', raw=response.content)
transfer_credit (line ~434-436)	three soup.find('input', ...) lines for name, token, toUserId	three calls to _find_input_value with distinct context values
get_register_link (line ~402-405)	soup.find('form', {'id': 'qrCodeForm'}).find('a')['href']	dedicated handling: dump on miss, raise informative ScraperError (this one is not an input lookup)

The get_user_credit method's text-parsing path (soup.find('table', {...}).find(text=...).parent.parent.find_all('td')[2].text) is structurally different — it doesn't fit the input-value helper. Its existing try/except block already returns 0 on failure. We extend it: in the except, dump the response (unconditional, not commented) and print the path so the operator sees it. We do not change the return contract (still returns 0 on failure) because callers depend on that.

get_register_link (the QR code form) is the other non-input case. We add an explicit if soup.find(...) is None: self._dump_html(...); raise ScraperError(...) block — short, no new helper.

Why no token caching

Struts2 anti-CSRF tokens (the struts.token.name/token pair) are typically regenerated per form-load: load form → token A → submit → token A is invalidated → load form → token B. Re-using token A for a second submit will fail with "Invalid Token". The current code re-fetches per operation precisely because of this contract. Caching might work if this Struts deployment treats the token as session-scoped, but verifying that requires the operator to test it manually against cm99.net, which:

• Costs real cm99.net traffic
• Is not in this scope
• Saves at most ~100ms per operation, which is not the bot's bottleneck

If a future cycle wants to optimize bot throughput, it can characterize the token contract first. For now: skip.

Files Created / Modified

File	Operation
app/cm_bot.py	Modify — add ScraperError, _dump_html, _find_input_value; convert five call sites + the get_register_link and get_user_credit failure paths.
tests/test_cm_bot_scraper.py	Create — unit tests for the three new helpers.
.gitignore	Already covers logs/ (existing entry). No change needed.

No new dependencies. No new tests for existing call sites — those are integration-level and would require live cm99.net or HTML fixtures we don't currently maintain.

Verification

1. Unit tests pass. .venv/bin/python -m unittest tests.test_cm_bot_scraper -v exercises:
   • _find_input_value returns value when input exists.
   • _find_input_value raises ScraperError when input is missing, and the error message contains the saved path.
   • _find_input_value raises ScraperError when input has no value attribute.
   • _dump_html creates the directory if missing and writes the bytes verbatim.
   • _dump_html returns the path it wrote to.
2. Whole suite still green. .venv/bin/python -m unittest tests.test_debug_enabled tests.test_bot_cli tests.test_cm_bot_scraper -v → OK.
3. Real-call smoke (deferred to operator). Trigger a bot operation against cm99.net. If anything fails, an HTML file appears under logs/scraper-failures/<context>-<timestamp>.html and the container logs print the path. Existing happy paths are unchanged.

Risk

Low.

• The new helpers' behavior is byte-equivalent to the existing code on the happy path (input exists with value). The value is returned exactly the same way.
• Failure mode changes from TypeError: 'NoneType' object is not subscriptable to ScraperError: <context>: input[name='token'] missing or has no value (response saved to logs/...). Existing exception handlers in cm_bot_hal.py (except Exception as e: ..., except: ...) catch both equally.
• Disk usage: failing HTML responses go into logs/scraper-failures/. cm99.net pages are ~10–30KB each. At a few failures per day this is negligible. If cm99.net misbehaves badly (every request fails) the operator has bigger problems anyway and will see the spam in container logs.

Out-of-Scope Follow-Ups

• Struts token caching after characterizing the contract. Not now.
• Periodic cleanup of logs/scraper-failures/ (e.g., a find logs/ -name '*.html' -mtime +14 -delete cron). YAGNI; revisit if disk usage becomes a concern.
• HTML fixture-based regression tests for the bot operations themselves. Useful long-term; large enough scope to be its own cycle.