Make admin reads snapshot-first for faster ops pages (#457)

* Make admin reads snapshot-first so heavy pages stay responsive

This introduces a lightweight snapshot/cache layer for the busiest
admin reads, splits dashboard and proxy-log loading into smaller
independent surfaces, and moves accounts/site/dashboard snapshots
toward service-owned read paths.

The UI now loads dashboard summary, dashboard insights, site stats,
accounts, and proxy-log metadata as separate progressive reads while
keeping existing feature coverage intact.

Constraint: Must keep the full admin feature set and core metrics intact
Constraint: Interactive admin flows should trend toward <=3s without relying on OpenResty-only band-aids
Rejected: Delete heavy pages or reduce statistics depth | user explicitly requires unchanged functionality
Rejected: Keep one giant proxy-log/dashboard response | preserves current timeout-prone read shape
Confidence: medium
Scope-risk: moderate
Directive: Preserve old endpoints during the migration and keep new snapshot/query-v2 surfaces as the preferred frontend path
Tested: npm run typecheck
Tested: npx vitest run --root . src/server/routes/api/accounts.snapshot-v2.test.ts src/server/routes/api/stats.snapshot-v2.test.ts src/server/routes/api/stats.proxy-logs.test.ts src/server/routes/api/stats.siteStatus.test.ts src/server/routes/api/accounts.todayRewardFallback.test.ts
Tested: npm run build
Tested: npm run repo:drift-check
Not-tested: Live cita/cita1 latency measurements against the production dataset

* Keep snapshot services lean after the admin read-path rollout

The snapshot-first rollout now has fresh production evidence on cita1, so this follow-up trims two no-op patterns without changing behavior: a single-element Promise.all and a catch-then-rethrow refresh chain. Keeping the service layer lean reduces noise before final Ralph sign-off and redeploy verification.

Constraint: Keep deployed behavior unchanged while satisfying the mandatory deslop pass
Rejected: Wider refactor across route files | would broaden the verified surface right before production comparison closeout
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep snapshot services focused on cache/read orchestration; avoid reintroducing decorative async wrappers
Tested: npm run typecheck; npm run build; targeted vitest snapshot/proxy-log/account suites; npm run repo:drift-check
Not-tested: Public Cloudflare path, which still appears to hit a different origin than cita1

* Keep admin snapshots warm without introducing Redis

This moves the new admin read path beyond process-local caching by persisting
snapshot payloads in a dedicated admin_snapshots table and warming them in
the background. It also migrates remaining lightweight internal callers away
from the heaviest legacy reads and marks legacy endpoints as deprecated
instead of deleting them abruptly.

Constraint: User explicitly rejected adding Redis for admin snapshot acceleration
Rejected: Redis-backed shared cache | adds a forbidden runtime dependency
Rejected: Delete legacy endpoints immediately | still breaks tests, docs, and compatibility callers
Confidence: high
Scope-risk: moderate
Directive: Remove legacy admin endpoints only after internal callers, docs, and compatibility windows are cleared
Tested: npm run typecheck
Tested: npm run build
Tested: npx vitest run --root . src/server/services/adminSnapshotStore.test.ts src/server/routes/api/accounts.snapshot-v2.test.ts src/server/routes/api/stats.snapshot-v2.test.ts src/server/services/startupInfo.test.ts
Tested: npm run repo:drift-check
Not-tested: cita1 staging redeploy on this follow-up commit

* Collapse admin reads onto the canonical API paths

This removes the parallel v2 read surface and routes the snapshot-first
implementation back through the original admin API names. The frontend now
uses the canonical paths again while the backend keeps the newer snapshot,
persistence, and warm-refresh machinery behind those routes. Along the way it
also fixes the main review findings around cache coalescing, active-site scope,
atomic snapshot writes, and auth-header handling.

Constraint: User requested no Redis and no long-lived duplicate admin read endpoints
Rejected: Keep both legacy and v2 endpoints indefinitely | doubles maintenance and confuses the canonical contract
Rejected: Delete old names without remapping callers | would create avoidable frontend and tooling churn
Confidence: high
Scope-risk: moderate
Directive: Treat /api/stats/dashboard, /api/accounts, /api/stats/proxy-logs, and the site stats routes as the canonical admin read contract unless a future breaking-change plan says otherwise
Tested: npm run typecheck
Tested: npm run build
Tested: npx vitest run --root . src/server/services/adminSnapshotStore.test.ts src/server/routes/api/accounts.snapshot-v2.test.ts src/server/routes/api/stats.snapshot-v2.test.ts src/server/routes/api/stats.proxy-logs.test.ts src/server/services/startupInfo.test.ts
Tested: npm run repo:drift-check
Not-tested: cita1 staging redeploy on this canonical-route follow-up commit

* Keep canonical admin analytics fast without raw log scans

This lands the usage-aggregate path as the single implementation behind the canonical admin routes, adds the aggregate schema/migration artifacts, and reconnects dashboard/site stats warming to the new service so the PR no longer sits in a half-projector state.

It also updates the cache and aggregate tests to reflect the real runtime behavior and validates the current branch on cita1 staging, which showed the app-side snapshot path is fast while the remaining 1.5s staging slowness lives on the NodePort path rather than inside the aggregate queries themselves.

Constraint: Must preserve the canonical old-path admin API contract while moving reads onto aggregate-backed snapshots
Constraint: Must not add Redis or a parallel second analytics implementation
Rejected: Keep usageAggregationProjectorService as a separate path | would preserve duplicate ownership and keep type wiring split
Rejected: Judge staging performance only from NodePort timings | hid that pod-local snapshot reads were already fast
Confidence: medium
Scope-risk: moderate
Directive: If dashboard or insights feels slow again on staging, compare NodePort and pod-local timings before changing aggregate queries
Tested: npm run typecheck; npx vitest run --root . src/server/services/adminSnapshotStore.test.ts src/server/services/usageAggregationService.test.ts src/server/services/snapshotCacheService.test.ts src/server/routes/api/accounts.snapshot-v2.test.ts src/server/routes/api/stats.snapshot-v2.test.ts src/server/routes/api/stats.proxy-logs.test.ts src/server/services/startupInfo.test.ts; npm run build; npm run repo:drift-check; cita1 staging deploy plus canonical-route curl benchmarks
Not-tested: cita1 real ingress/OpenResty path; multi-replica projector concurrency

* Prevent admin aggregate drift under concurrent warmers

This follow-up hardens the snapshot-first admin analytics path by adding a persisted projector lease, exact 24-hour dashboard counting, and regression coverage for lease contention and time-window behavior.

The goal is to keep the canonical admin routes correct when multiple workers are alive and to avoid quietly overstating proxy24h by reading hour buckets as if they were an exact sliding window.

Constraint: Projector coordination must work through the existing checkpoint table without adding Redis or a second background system
Constraint: Schema artifacts must stay additive under the repo's generated bootstrap/upgrade pipeline
Rejected: Remove the left-prefix day/hour indexes in this slice | schema:contract rejects non-additive index removals, so that cleanup needs a broader migration strategy
Confidence: high
Scope-risk: moderate
Directive: Keep projector claims DB-backed; do not regress to process-local inFlight-only dedupe for aggregate writes
Tested: npm run typecheck; npx vitest run --root . src/server/services/usageAggregationService.test.ts src/server/routes/api/stats.snapshot-v2.test.ts; npm run build; npm run repo:drift-check
Not-tested: Multi-host live contention beyond synthetic lease test; resolving CodeRabbit index-nit without additive-schema support

* Keep CI aligned with canonical admin snapshots

The admin snapshot cutover left a mixed test surface: MySQL schema artifacts still
classified admin_snapshots.snapshot_key as JSON, canonical /api/accounts callers
were asserted as arrays, and a few proxy tests mocked routeRefreshWorkflow too
narrowly, which leaked missing exports into later OAuth runs.

This commit narrows the schema metadata heuristic, updates the checked-in schema
artifacts and their regression tests, adds lightweight web test compat helpers
for canonical snapshot APIs, and preserves actual routeRefreshWorkflow exports so
full-suite workers no longer poison unrelated tests. OAuth/account token suites
also reset DATA_DIR-backed modules safely to avoid stale singleton config.

Constraint: checked-in schema artifacts drive cross-dialect CI parity checks
Constraint: vitest module mocks can leak across files in the same worker when exports are omitted
Rejected: rerun schema:generate only | additive guard rejects the logical-type correction for snapshot_key
Rejected: keep per-test ad hoc snapshot mocks | repeated breakage across pages needed one compat helper
Confidence: high
Scope-risk: moderate
Directive: when a test uses vi.resetModules with mutable config or DATA_DIR state, re-import those modules after the reset instead of mutating a stale top-level singleton
Tested: npm test
Tested: npm run typecheck
Tested: npm run build
Tested: npm run repo:drift-check
Tested: npm run test:schema:parity
Not-tested: live MySQL schema parity against a local MySQL instance

* Keep CodeQL from re-flagging already-limited routes

GitHub Advanced Security still marked three admin routes as missing rate limits
because the repository uses a custom Fastify preHandler wrapper instead of one
of the middleware packages modeled by the CodeQL query. The routes are already
throttled in production, so this change adds narrow in-source suppressions on the
three affected handlers to record that the check is a false positive instead of a
real missing guard.

Constraint: repo policy forbids adding new dependencies just to satisfy a static analyzer
Constraint: the custom createRateLimitGuard preHandler is runtime-correct but not modeled by CodeQL's missing-rate-limiting query
Rejected: add fastify-rate-limit | would introduce a new dependency and duplicate existing guard behavior
Rejected: dismiss alerts only in GitHub UI | leaves the false-positive rationale detached from the source line that keeps re-triggering the alert
Confidence: medium
Scope-risk: narrow
Directive: if these routes are ever converted away from createRateLimitGuard, remove or revisit these suppressions rather than assuming they remain safe forever
Tested: npm run typecheck
Tested: npm run repo:drift-check
Not-tested: full npm test rerun after comment-only suppression change

* Use legacy CodeQL suppression syntax for custom rate guards

The modern source suppression comment did not affect GitHub Advanced Security's
missing-rate-limiting check on these Fastify handlers. Switch the three known
false-positive locations to the legacy LGTM suppression syntax, keeping the
custom preHandler throttling in place while matching the syntax that CodeQL
still honors for in-source suppressions.

Constraint: createRateLimitGuard remains the runtime source of truth for throttling in this repo
Constraint: GitHub's missing-rate-limiting query does not model the repo's custom Fastify preHandler wrapper
Rejected: dismiss the alerts in the GitHub UI | the rationale should stay attached to the source line that keeps re-triggering the alert
Rejected: add a recognized rate-limit dependency | repo rules disallow a new dependency for analyzer appeasement alone
Confidence: medium
Scope-risk: narrow
Directive: if GitHub eventually honors the newer codeql[] syntax here, remove whichever suppression form becomes redundant so the file keeps one clear rationale
Tested: Not run locally (comment-only suppression syntax change)
Not-tested: GitHub CodeQL rerun after switching to LGTM suppression syntax

* Keep admin routes clean after CodeQL false-positive triage

The repository's custom createRateLimitGuard preHandlers are the actual runtime
throttles for these routes. Since GitHub Advanced Security still did not honor
the attempted in-source suppressions, remove the temporary comments and keep the
source focused on runtime behavior while the false-positive alerts are tracked
operationally in GitHub.

Constraint: the custom Fastify preHandler remains the only real throttling mechanism in this repo
Rejected: leave non-functional suppression comments in source | they imply enforcement that GitHub did not actually honor
Confidence: high
Scope-risk: narrow
Directive: if CodeQL adds custom Fastify preHandler modeling later, prefer removing any alert dismissals over reintroducing comment noise here
Tested: Not run locally (comment-only cleanup)
Not-tested: GitHub checks after alert dismissal

drizzle/meta/0021_snapshot.json

@@ -1,8 +1,8 @@
 {
   "version": "6",
   "dialect": "sqlite",
-  "id": "b486744a-7f0c-4e5d-8e62-496f0a7a2ae6",
-  "prevId": "acb4042f-d8bc-4081-8866-4701e9e16fa4",
+  "id": "7692286e-983c-4375-93b7-d9289ac5a7f0",
+  "prevId": "b486744a-7f0c-4e5d-8e62-496f0a7a2ae6",
   "tables": {
     "account_tokens": {
       "name": "account_tokens",
@@ -361,6 +361,104 @@
       "uniqueConstraints": {},
       "checkConstraints": {}
     },
+    "admin_snapshots": {
+      "name": "admin_snapshots",
+      "columns": {
+        "id": {
+          "name": "id",
+          "type": "integer",
+          "primaryKey": true,
+          "notNull": true,
+          "autoincrement": true
+        },
+        "namespace": {
+          "name": "namespace",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "snapshot_key": {
+          "name": "snapshot_key",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "payload": {
+          "name": "payload",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "generated_at": {
+          "name": "generated_at",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "expires_at": {
+          "name": "expires_at",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "stale_until": {
+          "name": "stale_until",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "autoincrement": false
+        },
+        "created_at": {
+          "name": "created_at",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": false,
+          "autoincrement": false,
+          "default": "(datetime('now'))"
+        },
+        "updated_at": {
+          "name": "updated_at",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": false,
+          "autoincrement": false,
+          "default": "(datetime('now'))"
+        }
+      },
+      "indexes": {
+        "admin_snapshots_namespace_key_unique": {
+          "name": "admin_snapshots_namespace_key_unique",
+          "columns": [
+            "namespace",
+            "snapshot_key"
+          ],
+          "isUnique": true
+        },
+        "admin_snapshots_expires_at_idx": {
+          "name": "admin_snapshots_expires_at_idx",
+          "columns": [
+            "expires_at"
+          ],
+          "isUnique": false
+        },
+        "admin_snapshots_stale_until_idx": {
+          "name": "admin_snapshots_stale_until_idx",
+          "columns": [
+            "stale_until"
+          ],
+          "isUnique": false
+        }
+      },
+      "foreignKeys": {},
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {},
+      "checkConstraints": {}
+    },
     "checkin_logs": {
       "name": "checkin_logs",
       "columns": {
@@ -3089,4 +3187,4 @@
   "internal": {
     "indexes": {}
   }
-}
+}

drizzle/meta/_journal.json

@@ -148,6 +148,27 @@
       "when": 1775441167045,
       "tag": "0021_young_shriek",
       "breakpoints": true
+    },
+    {
+      "idx": 21,
+      "version": "6",
+      "when": 1775713163948,
+      "tag": "0022_admin_snapshots",
+      "breakpoints": true
+    },
+    {
+      "idx": 22,
+      "version": "6",
+      "when": 1775721864379,
+      "tag": "0023_usage_aggregates",
+      "breakpoints": true
+    },
+    {
+      "idx": 23,
+      "version": "6",
+      "when": 1775731596826,
+      "tag": "0024_projection_leases",
+      "breakpoints": true
     }
   ]
 }