For 3k+ row deployments, returning the full table in one shot is the
bottleneck — the JSON payload alone is hundreds of KB and the client
mounts thousands of EditableCell instances on every visit. Pagination
with auto-fetch on scroll shrinks both the wire payload and the initial
render to a single page (200 rows).
Server (app/cm_api.py):
- /acc/ and /user/ accept ?limit, ?offset, ?prefix, ?dir, plus ?sort on
/user/ (f_username | last_update_time). Defaults: limit=200 (capped at
1000), offset=0, dir=desc.
- ORDER BY done in SQL with prefix-priority: rows whose username starts
with the configured CM_PREFIX_PATTERN come first, then asc/desc by the
sort column. The 'dir' value is whitelisted to ASC|DESC before string
interpolation; everything else goes through parameterised binding.
- Schema verification (verify_tables_once) deferred to first request via
a Flask before_request hook — keeps create_app() free of MySQL touches
so unit tests + gunicorn preload still work without a live DB.
Web client:
- web/lib/api.ts: getAccountsPage / getUsersPage return { rows, hasMore }.
hasMore = (rows.length === PAGE_SIZE), so the client knows when to
stop fetching. Each page is its own Next.js cache entry (the URL is
the cache key) — caching from the previous commit still applies.
- web/app/actions.ts: loadMoreAccounts / loadMoreUsers Server Actions
for next-page requests; refreshAccounts / refreshUsers force-evict the
cache via revalidateTag before refetching page 1.
- web/app/page.tsx + users/page.tsx: only fetch the first page now.
- web/components/{accounts,users}-table.tsx: rewrote state model. Rows
accumulate as the user scrolls. An IntersectionObserver on a sentinel
div near the bottom triggers loadMore when it enters the viewport
(300px rootMargin so the next page starts loading before the user
reaches the end). useOptimistic wraps the accumulated rows for in-
flight edits; on success the row is committed locally so the change
survives even though we no longer router.refresh.
- Sort toggle now refetches from page 1 with the new dir/sort param.
Local sort over a partial set would be inconsistent.
- Mutations: delete filters from local state; create + refresh both
reset to page 1 so the row appears in its sorted position.
- Header count shows '<loaded>+' when more pages exist so the operator
knows what they're seeing isn't the full table.
Removed AutoRefresh:
- web/app/layout.tsx no longer mounts AutoRefresh.
- web/components/auto-refresh.tsx deleted.
- Reason: router.refresh every 30s would yank the user back to page 1
every time, losing scroll position and accumulated rows. Manual
Refresh button replaces it (now wired to refreshAccounts/refreshUsers
which evict cache + refetch).
Tests: deferred verify_tables_once() means tests.test_bot_cli's
CreateAppFactoryTests pass without DB env vars again. All 38 existing
tests pass.
Eliminates the per-request DB hit when:
- A user opens a tab they've recently visited (within 30s)
- The 30s AutoRefresh fires while no mutations have happened
- Multiple browser tabs are open and switch between Accounts/Users
- A passing-by request races another request for the same data
How it works:
- web/lib/api.ts: fetchApi() now forwards the next.revalidate/tags
options to Next.js's data cache. getAccounts/getUsers tag their
responses ('accounts'/'users') with a 30-second freshness window.
- web/app/actions.ts: every mutation (update/create/delete X 2 tables)
calls revalidateTag() so the next GET for that table bypasses the
cache and re-reads from MySQL. Stale data never lingers after a write.
The cache lives in the cm-web Node process (per worker). For our
2-worker setup that's at most 2 cached copies; the next AutoRefresh
tick after the 30s window expires triggers exactly one DB read per
worker. If the operator manually clicks Refresh, that's a router.refresh
which also re-fetches.
Tradeoffs:
- External DB writes (e.g., the cm99.net monitor inserting a row) won't
appear in the dashboard until the 30s window elapses or a mutation
happens. The previous behavior had a 30s ceiling too (auto-refresh
interval), so the perceived freshness is unchanged.
- Memory: each cached payload is a few KB to a few hundred KB. Trivial.
If you want stricter freshness later, drop CACHE_REVALIDATE_SECONDS in
web/lib/api.ts. If you want pagination on top of this, the cache key
becomes per-URL automatically, so /acc/?offset=200 caches separately
from /acc/?offset=0 — no further work needed.
In @simplewebauthn/server v11 the JSON response and transport types are
no longer re-exported from the server package — they live in the sibling
@simplewebauthn/types package. Adds the dep and switches the imports.
Previously the session cookie used Secure=NODE_ENV==='production', and the
dev override still runs the standalone build with NODE_ENV=production, so
the cookie was unreachable from phone-on-LAN testing over HTTP. Switching
to CM_DEBUG lets dev (CM_DEBUG=true) drop the Secure flag while keeping
prod (CM_DEBUG=false) safe.
Also wires CM_AGENT_ID/CM_AGENT_PASSWORD/CM_DEBUG into the web-next
service env block so the login Server Action can compare against them.
The Route Handler proxy and hash mapping are gone. Browser never
hits a JSON endpoint: data reads happen in React Server Components
fetching api-server:3000 server-side; mutations (B2) will use
Next.js Server Actions. Zero public API surface to scrape or
enumerate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
