feat: merge opus comfyui and login updates

openspec/changes/comfyui-multi-tenant-api-only/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-05-20

openspec/changes/comfyui-multi-tenant-api-only/design.md

@@ -0,0 +1,102 @@
+## Context
+
+Today the Rust backend (`bosapi v1.3.0`) spawns a single embedded ComfyUI subprocess at boot via `services/comfyui_process.rs::ComfyUIProcess::new(...)` and stores it in `AppState.comfyui: Arc<ComfyUIProcess>`. All `/api/video-agent/*` handlers and any future `/comfy/*` traffic share this one process — so workflows, queue items, custom node state, and uploaded media leak between users. We need to support 100+ users with full per-user isolation, but we have no GPU (we use the BoosAPI text/image/TTS/video endpoints exclusively), so per-user instances are cheap (~1–2 GB resident with the API-only profile).
+
+The user already runs ComfyUI in Docker as image `bosapi-comfyui:ready` and has accepted the "立刻可用 + 完全隔离" path: per-user on-demand container, with K3S as the v2 target. They added two late constraints we must honor:
+
+1. Cap each container at **2 CPU / 2 GB RAM** (`--cpus=2 --memory=2g --memory-swap=2g`).
+2. The existing `/api/video-agent/*` route surface is the in-app ComfyUI consumer; it must keep working for the same frontend with no contract change.
+
+Auth is the existing JWT cookie. Users are identified by `user_id` from the cookie's claims. There is no per-org limit yet — only a single global cap.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Per-user ComfyUI isolation: workflows, queue, history, and uploaded files MUST NOT cross users.
+- Lazy startup: a container exists only when the user is actively working; reaped after **60 minutes** idle.
+- Hard ceiling: at most **30 concurrent containers**, hard-capped to bound memory at ~60 GB.
+- Per-container resource cap: **2 CPU / 2 GB RAM** (memory + swap).
+- Persistent custom workflows and outputs at `/data/comfyui-users/<user_id>/`.
+- Reuse the existing `/api/video-agent/*` route surface unchanged from the frontend's perspective.
+- New `/comfy/*` HTTP+WebSocket reverse proxy with cookie auth (no username in URL).
+- Be K3S-portable: the container backend lives behind a `ContainerRuntime` trait so v2 can swap Docker for K3S with no caller changes.
+- Built-in ComfyUI OpenAI nodes work via the `BOOSAPI_BUILTIN_KEY` env-var fallback (already patched in `_helpers.py`; bake into image).
+
+**Non-Goals:**
+- GPU support, local model weights, training. We are API-only.
+- Per-organization quotas or per-user concurrency budgets — single global cap of 30 only.
+- K3S backend implementation. v1 ships Docker only; the trait is the only K3S investment.
+- Migrating away from `bosapi_comfy_nodes.py`. It coexists with the patched internal nodes.
+- Replacing the existing dev-time `docker-compose.comfyui.yml`. Production stops using it; dev keeps it.
+
+## Decisions
+
+### 1. Per-user on-demand Docker container (vs shared+key-injection or per-node-key)
+**Chosen:** spawn `bosapi-comfyui:multi-tenant` once per active user via `docker run`. **Rationale:** complete filesystem and queue isolation; no need to rewrite ComfyUI's WS protocol to mux users; the user's own framing was "立刻可用 + 完全隔离 ... 后期用K3S". **Alternatives considered:** (a) shared instance with per-request key injection — leaks queue/history across users, requires a reverse-engineered WS proxy that rewrites `client_id`, fragile; (b) per-node API-key — loses access to ComfyUI's built-in OpenAI nodes and prebuilt templates the user explicitly wants kept ("内置的节点和模板都很好用").
+
+### 2. Pool sizing: 30 hard cap, 60-minute idle reap
+**Chosen:** `MAX_CONCURRENT=30`, `IDLE_TIMEOUT=60min`. **Rationale:** 30 × 2 GB = 60 GB resident, fits a single mid-size box; 60 min covers a typical creative session without forcing repeated cold starts. Beyond 30, the pool returns `503` with `Retry-After`. **Alternative considered:** unbounded with LRU evict — but evicting a busy queue mid-render is worse UX than asking the next user to wait.
+
+### 3. URL routing: `/comfy/*` at host root, identity from JWT cookie
+**Chosen:** mount a reverse-proxy router at `/comfy/*` that reads the JWT cookie, resolves the user's container, and proxies HTTP+WS to `127.0.0.1:<port>`. **Rationale:** ComfyUI's HTML/JS asset paths assume root-relative URLs and break under subpath rewriting; putting the user_id in the URL path leaks it and complicates sharing/auth. **Alternative considered:** subdomain-per-user (`<user>.comfy.boos.lat`) — needs wildcard TLS and DNS, overkill for v1.
+
+### 4. Container backend behind `ContainerRuntime` trait
+**Chosen:**
+```rust
+#[async_trait]
+pub trait ContainerRuntime: Send + Sync {
+    async fn start(&self, spec: ContainerSpec) -> Result<ContainerHandle>;
+    async fn stop(&self, id: &str) -> Result<()>;
+    async fn is_alive(&self, id: &str) -> Result<bool>;
+}
+```
+v1 ships `DockerRuntime` (shells out to `docker` or talks to `/var/run/docker.sock`). v2 adds `K3sRuntime`. **Rationale:** the user has already picked K3S as v2; constraining now costs little and saves a rewrite. **Alternative considered:** hard-code `bollard` calls into the pool — couples lifecycle to Docker, rejected.
+
+### 5. Resource limits per container
+**Chosen:** `--cpus=2 --memory=2g --memory-swap=2g`, plus `--pids-limit=512`, `--read-only` rootfs with `--tmpfs /tmp:size=512m`, `--cap-drop=ALL`, `--security-opt=no-new-privileges`. Volume mounts: `/data/comfyui-users/<user_id>/input`, `/output`, `/temp`, `/user`, `/custom_nodes_user`. **Rationale:** user explicitly said "Docker的CPU RAM都要做限制 给个2C2G". The hardening flags follow defense-in-depth; ComfyUI does not need privileges or rootfs writes beyond the mounted volumes.
+
+### 6. Idle reaper as a background tokio task
+**Chosen:** a single tokio task ticks every 60 s, scans `pool.containers`, and stops any whose `last_used_at` is > 60 min ago. State recorded in-memory; a crash drops it but containers self-stop on `--rm` exit anyway (we do `docker stop`, not `docker kill`, so ComfyUI flushes). **Alternative considered:** Docker's own `--stop-timeout` + cron — split control plane between Rust and shell, harder to reason about.
+
+### 7. Refactor of `/api/video-agent/*` (preserve frontend contract)
+**Chosen:** every handler in `routes/video_agent.rs` extracts `user_id` from the JWT cookie, calls `state.comfyui_pool.get_or_start(user_id).await?`, and forwards to that handle's `start/stop/restart/status` or HTTP client. Status responses gain `containerState: "starting" | "ready" | "stopped"` so the frontend can render the new full-screen loading overlay. The existing JSON keys remain. **Alternative considered:** introduce a v2 namespace `/api/v2/video-agent/*` — gratuitous churn on the frontend.
+
+### 8. AppState rename
+**Chosen:** `pub comfyui_pool: Arc<ComfyUIPool>` replaces `pub comfyui: Arc<ComfyUIProcess>`. The old `ComfyUIProcess` type is **kept** — it's now the implementation detail used inside `DockerRuntime` to model the in-container ComfyUI subprocess (start/stop, port readiness probe, log buffer). `lib.rs` no longer constructs it directly; `DockerRuntime::start` does.
+
+### 9. Image build
+**Chosen:** new `docker/Dockerfile.comfyui` based on the user's existing `bosapi-comfyui:ready` lineage. Adds:
+- the patched `comfy_source/comfy_api_nodes/util/_helpers.py` (env-var fallback)
+- `bosapi_comfy_nodes.py` baked in (no host bind-mount, so it survives `--read-only`)
+- `ENTRYPOINT` reads `BOOSAPI_BUILTIN_KEY`, `BOOSAPI_BASE`, `BOOSAPI_KEY`, then exec's `python main.py --listen 0.0.0.0 --port 8188 --comfy-api-base "$BOOSAPI_BASE/api/comfyui-agent"`.
+Tag: `bosapi-comfyui:multi-tenant`. `build-and-run.sh` builds this image; the runtime spawns containers from it on demand.
+
+## Risks / Trade-offs
+
+- **Cold-start latency 10–20 s** → mitigated by full-screen loading UI with progress text; pool exposes `containerState` so the frontend can poll.
+- **30-container ceiling can reject users at peak** → 503 with `Retry-After: 60`; UI shows a friendly "服务繁忙" page. Monitor and raise the cap (or move to K3S) if it bites.
+- **Docker socket exposure** if the API runs in Docker → mount `/var/run/docker.sock` only, restrict to root in API container, never expose to user containers. Document for ops.
+- **Port pool exhaustion (8200–8299 = 100 slots, cap is 30)** → 70-port headroom is fine; allocator picks lowest-free; on container stop the port is released.
+- **Idle reap mid-render** → reap only when no active WS connections AND `last_used_at` > 60 min; bump `last_used_at` on every HTTP+WS frame. Acceptable false-negative rate.
+- **Persistent volume disk growth** → no quota in v1. Document; add a per-user disk cap (e.g. 5 GB via `du`+reject) only if it becomes a real problem.
+- **Built-in OpenAI nodes auth** depends on `BOOSAPI_BUILTIN_KEY` env reaching the container → enforce via `ContainerSpec.env` in the pool, not opt-in.
+- **K3S abstraction tax** → only the trait shape; we still have to design pod specs later. Acceptable.
+- **State loss on API restart** → containers keep running but the pool's in-memory map is gone. Mitigation: on startup, list `comfyui-user-*` containers, reconcile map. Tracked as a task.
+
+## Migration Plan
+
+1. Land `ContainerRuntime` trait + `DockerRuntime` + `ComfyUIPool` behind a feature flag `multi_tenant_comfy` (default off). The old `state.comfyui` singleton stays intact.
+2. Build `bosapi-comfyui:multi-tenant` image; smoke-test by hand with `docker run`.
+3. Add `/comfy/*` proxy and a single new `state.comfyui_pool` field alongside `state.comfyui`.
+4. Refactor `/api/video-agent/*` handlers to read from `state.comfyui_pool` when the flag is on, else fall back to `state.comfyui`.
+5. Toggle the flag on in staging; verify with two real users that workflows/uploads do not cross.
+6. Remove `state.comfyui` and the flag. `docker-compose.comfyui.yml` becomes dev-only.
+7. Document host requirements in `README` (Docker socket, `/data/comfyui-users/`, image build).
+
+**Rollback:** flip the feature flag off and redeploy. Containers from the multi-tenant path keep running until idle-reaped; no data loss because volumes are persistent.
+
+## Open Questions
+
+- Should the API container run with `docker:dind` or mount the host socket? Default to host socket; revisit if ops needs isolation.
+- Where do we publish the loading-state UI string set? Put it next to other i18n strings in the frontend repo.
+- Do we expose `/comfy/admin/*` for ops to introspect the pool? Out of scope for v1; add later if needed.

openspec/changes/comfyui-multi-tenant-api-only/proposal.md

@@ -0,0 +1,43 @@
+## Why
+
+Today every user of `/api/video-agent/*` and `/comfy/*` shares one global `Arc<ComfyUIProcess>` singleton spawned at boot. That means workflows, queue, custom nodes, and uploaded assets are visible across users — unsuitable for the 100+ user multi-tenant deployment we are moving toward. We need per-user isolation now, with an architecture that can move to K3S later without a second rewrite.
+
+## What Changes
+
+- **BREAKING (internal API only):** Replace the `state.comfyui: Arc<ComfyUIProcess>` singleton with `state.comfyui_pool: Arc<ComfyUIPool>`. All existing `/api/video-agent/*` handlers refactor to look up (or lazily start) the caller's container via the pool.
+- Introduce per-user on-demand ComfyUI containers. One container per authenticated user, started on first `/comfy/*` or `/api/video-agent/*` request, reaped after **60 minutes idle**.
+- Cap the pool at **30 concurrent containers** (≈60 GB RAM at 2 GB each). Requests beyond that queue with a clear "service busy" response and full-screen loading UI hint.
+- Each container limited to **2 CPU / 2 GB RAM** (`--cpus=2 --memory=2g --memory-swap=2g`), bound to a port from the **8200–8299** pool, named `comfyui-user-<user_id>`.
+- Persistent per-user volume mounted at `/data/comfyui-users/<user_id>/` so custom workflows, inputs, and outputs survive container restarts.
+- Add a reverse-proxy route family `/comfy/*` (HTTP + WebSocket) that routes the authenticated user to their own container. **No username in URL** — identity comes from the existing JWT cookie. ComfyUI's HTML/asset paths assume root, so we proxy at the host level rather than under a subpath.
+- Bake the existing `comfy_source/comfy_api_nodes/util/_helpers.py` env-var fallback patch into the container image so ComfyUI's built-in OpenAI nodes pick up `BOOSAPI_BUILTIN_KEY` automatically.
+- Abstract the container backend behind a `ContainerRuntime` trait. v1 ships `DockerRuntime`. v2 (when we move to K3S) adds `K3sRuntime` without touching pool, proxy, or route code.
+- Wire pool startup, idle reaper, and proxy into the existing axum app and `build-and-run.sh`.
+
+## Capabilities
+
+### New Capabilities
+- `comfyui-multi-tenant`: per-user ComfyUI container lifecycle (start, idle reap, max-concurrency, port allocation, persistent volumes).
+- `container-runtime`: backend-agnostic container abstraction (`ContainerRuntime` trait + `DockerRuntime` impl) so K3S can be added later without rewriting callers.
+- `comfyui-proxy`: cookie-authenticated reverse proxy at `/comfy/*` (HTTP + WebSocket) that maps the request to the caller's container.
+- `video-agent-per-user`: refactor of the existing `/api/video-agent/*` handlers to use the per-user pool instead of the global singleton, preserving the current frontend contract.
+
+### Modified Capabilities
+<!-- None — no specs/ directory exists yet, so all four capabilities are net-new. -->
+
+## Impact
+
+- **Code touched (Rust):**
+  - `src/services/comfyui_process.rs` — keep as the in-container process model used by `DockerRuntime`, no longer spawned directly by `lib.rs`.
+  - `src/services/` — new `comfyui_pool.rs`, `container_runtime.rs`, `docker_runtime.rs`.
+  - `src/routes/video_agent.rs` — every handler switches from `state.comfyui` to `state.comfyui_pool.get_or_start(user_id)`.
+  - `src/routes/` — new `comfy_proxy.rs` mounting `/comfy/*` (HTTP + WS).
+  - `src/lib.rs` — `AppState` field rename, pool construction, idle-reaper task spawn.
+- **Infra:**
+  - New `docker/Dockerfile.comfyui` baking the `_helpers.py` patch and `bosapi_comfy_nodes.py` into `bosapi-comfyui:multi-tenant`.
+  - `build-and-run.sh` builds the image; the Rust binary itself launches/stops containers at runtime (no docker-compose).
+  - Host paths: `/data/comfyui-users/<user_id>/` (persistent), `/var/run/docker.sock` mounted into the API container if API runs in Docker.
+- **Auth:** Existing JWT cookie used as-is. New `/comfy/*` proxy reads it; existing `/api/video-agent/*` already does.
+- **Frontend:** No contract change for `/api/video-agent/*`. New full-screen loading UI shown during the 10–20 s container startup; backend exposes a status endpoint the UI can poll.
+- **Removals:** Old `docker-compose.comfyui.yml` becomes dev-only; production no longer uses it. The single global `state.comfyui` field disappears.
+- **Out of scope:** GPU support (we are API-only), per-org quotas (single global cap of 30 only), K3S backend implementation (v2 task).

openspec/changes/comfyui-multi-tenant-api-only/specs/comfyui-multi-tenant/spec.md

@@ -0,0 +1,78 @@
+## ADDED Requirements
+
+### Requirement: Per-user container lifecycle
+The system SHALL maintain at most one ComfyUI container per authenticated user. The container SHALL be started lazily on the user's first request to `/comfy/*` or `/api/video-agent/*` and SHALL be reaped after 60 minutes of inactivity.
+
+#### Scenario: First request starts the container
+- **WHEN** an authenticated user with no running container makes a request to `/comfy/*`
+- **THEN** the system starts a container named `comfyui-user-<user_id>`, allocates a port from 8200–8299, records `last_used_at = now`, and forwards the request once the container's HTTP port is reachable
+
+#### Scenario: Subsequent request reuses the container
+- **WHEN** the same user makes another request within 60 minutes
+- **THEN** the system reuses the existing container, updates `last_used_at = now`, and forwards the request without restarting
+
+#### Scenario: Idle container is reaped
+- **WHEN** a container's `last_used_at` is older than 60 minutes AND it has no active WebSocket connections
+- **THEN** the system stops the container via the runtime's `stop` method and releases its port back to the pool
+
+### Requirement: Concurrency cap
+The system SHALL refuse to start a new container when 30 are already running.
+
+#### Scenario: 31st user is rejected
+- **WHEN** 30 containers are running and a 31st distinct user requests `/comfy/*`
+- **THEN** the system responds with HTTP 503 and a `Retry-After: 60` header, and does NOT evict any existing container
+
+#### Scenario: Cap frees on idle reap
+- **WHEN** any of the 30 containers is reaped
+- **THEN** the next pending user request that arrives starts a fresh container in the freed slot
+
+### Requirement: Resource limits
+Each container SHALL run with `--cpus=2`, `--memory=2g`, `--memory-swap=2g`, `--pids-limit=512`, `--read-only` rootfs, `--cap-drop=ALL`, and `--security-opt=no-new-privileges`.
+
+#### Scenario: Container respects CPU and memory caps
+- **WHEN** a new container is started
+- **THEN** `docker inspect` shows `HostConfig.NanoCpus = 2000000000`, `HostConfig.Memory = 2147483648`, `HostConfig.MemorySwap = 2147483648`
+
+#### Scenario: Container has no extra capabilities
+- **WHEN** a new container is started
+- **THEN** `docker inspect` shows `HostConfig.CapDrop` contains `ALL` and `HostConfig.SecurityOpt` contains `no-new-privileges`
+
+### Requirement: Persistent per-user volumes
+Each container SHALL mount the host directory `/data/comfyui-users/<user_id>/` so that custom workflows, inputs, and outputs survive container restarts.
+
+#### Scenario: Workflow saved in one session is visible after restart
+- **WHEN** a user saves a workflow file to ComfyUI's `user/` directory, then triggers a container restart (idle reap + new request)
+- **THEN** the workflow file is still present in the new container's `user/` directory
+
+#### Scenario: Volume is per-user
+- **WHEN** user A saves a file and user B starts their own container
+- **THEN** user B's container does NOT see user A's file
+
+### Requirement: Built-in API key injection
+Each container SHALL receive `BOOSAPI_BUILTIN_KEY`, `BOOSAPI_BASE`, and `BOOSAPI_KEY` as environment variables so ComfyUI's built-in OpenAI nodes (via the patched `_helpers.py`) and the `bosapi_comfy_nodes.py` custom nodes both authenticate successfully.
+
+#### Scenario: Internal OpenAI node authenticates
+- **WHEN** a workflow uses a built-in ComfyUI OpenAI node and the container has `BOOSAPI_BUILTIN_KEY` set
+- **THEN** the node's outbound request to the BoosAPI shim includes `Authorization: Bearer <BOOSAPI_BUILTIN_KEY>`
+
+#### Scenario: Custom BoosAPI node authenticates
+- **WHEN** a workflow uses `BoosAPITextGen` from `bosapi_comfy_nodes.py` and the container has `BOOSAPI_KEY` set
+- **THEN** the node's request to `BOOSAPI_BASE` includes `Authorization: Bearer <BOOSAPI_KEY>`
+
+### Requirement: Container state observable to callers
+The pool SHALL expose each container's state as `starting`, `ready`, or `stopped` so the frontend can render a loading UI during cold start.
+
+#### Scenario: Status reflects starting then ready
+- **WHEN** a user's first request triggers a container start
+- **THEN** `pool.state(user_id)` returns `starting` until the HTTP port is reachable, then `ready`
+
+#### Scenario: Status reflects stopped after reap
+- **WHEN** a container has been idle-reaped
+- **THEN** `pool.state(user_id)` returns `stopped`
+
+### Requirement: Reconcile on startup
+On API restart, the pool SHALL list all `comfyui-user-*` containers from the runtime and reconstruct its in-memory map (user_id, port, last_used_at).
+
+#### Scenario: Existing containers are adopted
+- **WHEN** the API process restarts while a `comfyui-user-42` container is still running
+- **THEN** the new pool's map contains an entry for user 42 with the original container's port, and the next request from user 42 reuses it

openspec/changes/comfyui-multi-tenant-api-only/specs/comfyui-proxy/spec.md

@@ -0,0 +1,45 @@
+## ADDED Requirements
+
+### Requirement: Cookie-authenticated reverse proxy at /comfy/*
+The system SHALL mount a reverse proxy at `/comfy/*` that resolves the caller's `user_id` from the existing JWT cookie and forwards both HTTP and WebSocket traffic to that user's container.
+
+#### Scenario: Anonymous request is rejected
+- **WHEN** a request to `/comfy/` arrives without a valid JWT cookie
+- **THEN** the proxy responds with HTTP 401 and does NOT touch the container pool
+
+#### Scenario: Authenticated HTTP request reaches the right container
+- **WHEN** user 7 requests `GET /comfy/api/object_info`
+- **THEN** the proxy forwards the request body, query, and headers to `http://127.0.0.1:<port_for_user_7>/api/object_info` and streams the response back unchanged
+
+#### Scenario: Authenticated WebSocket upgrade is proxied
+- **WHEN** user 7's browser opens `wss://host/comfy/ws?clientId=abc`
+- **THEN** the proxy upgrades the connection, opens a WS to `ws://127.0.0.1:<port_for_user_7>/ws?clientId=abc`, and pipes frames in both directions until either side closes
+
+### Requirement: No URL rewriting of asset paths
+The proxy SHALL forward request paths to the container with the `/comfy` prefix stripped and SHALL NOT rewrite response bodies.
+
+#### Scenario: Asset path is preserved
+- **WHEN** the browser requests `/comfy/extensions/core/foo.js`
+- **THEN** the proxy forwards `/extensions/core/foo.js` to the container, and the response body is returned byte-for-byte
+
+#### Scenario: ComfyUI's root-relative URLs work
+- **WHEN** the container returns HTML referencing `/scripts/app.js`
+- **THEN** the browser fetches `/comfy/scripts/app.js` (because the page was served under `/comfy/`) and the proxy resolves it correctly
+
+### Requirement: Cold-start handling
+If a request arrives while the user's container is `starting`, the proxy SHALL hold the request (up to 30 s) until the container reports `ready`, instead of returning 502.
+
+#### Scenario: Request waits for cold start
+- **WHEN** user 7's first request to `/comfy/` arrives and the container needs 12 s to start
+- **THEN** the proxy waits for the readiness signal and forwards the request once the container is ready, returning a normal 200
+
+#### Scenario: Cold-start timeout
+- **WHEN** a container fails to become ready within 30 s
+- **THEN** the proxy responds with HTTP 504 and the pool marks the container `failed` for the next idle scan
+
+### Requirement: Last-used timestamp updated on every frame
+Every HTTP request and WebSocket frame proxied to the container SHALL update `pool.containers[user_id].last_used_at = now`.
+
+#### Scenario: Long-running WS keeps container alive
+- **WHEN** a user holds a WS connection open for 90 minutes while exchanging frames every minute
+- **THEN** the container is not reaped at the 60-minute mark

openspec/changes/comfyui-multi-tenant-api-only/specs/container-runtime/spec.md

@@ -0,0 +1,34 @@
+## ADDED Requirements
+
+### Requirement: Backend-agnostic ContainerRuntime trait
+The system SHALL expose container lifecycle behind a single `ContainerRuntime` trait so the pool, proxy, and route handlers do NOT depend on Docker directly.
+
+#### Scenario: Pool uses only the trait
+- **WHEN** the source for `services/comfyui_pool.rs` is inspected
+- **THEN** it imports `ContainerRuntime` and `ContainerSpec` only — never `bollard`, `docker`, or `kube` types
+
+#### Scenario: Trait surface is sufficient for v1
+- **WHEN** a `DockerRuntime` is provided
+- **THEN** the pool can start containers, stop them, check liveness, and list existing `comfyui-user-*` containers using only the trait's methods
+
+### Requirement: DockerRuntime implementation
+The system SHALL ship a `DockerRuntime` that implements `ContainerRuntime` against the local Docker daemon, either via the host socket `/var/run/docker.sock` or the `docker` CLI.
+
+#### Scenario: Start spawns a container
+- **WHEN** `DockerRuntime::start(spec)` is called with `image=bosapi-comfyui:multi-tenant`, `name=comfyui-user-7`, port mapping, env, and volume mounts
+- **THEN** a container with that name exists, is running, and exposes the chosen host port
+
+#### Scenario: Stop removes the container
+- **WHEN** `DockerRuntime::stop("comfyui-user-7")` is called
+- **THEN** the container is stopped and removed (image used `--rm`-equivalent semantics) and its name becomes reusable
+
+#### Scenario: List finds existing user containers
+- **WHEN** `DockerRuntime::list_user_containers()` is called
+- **THEN** it returns one entry per running container whose name starts with `comfyui-user-`, including its port mapping
+
+### Requirement: K3S extension point reserved
+The trait SHALL be designed such that a future `K3sRuntime` implementing the same surface can replace `DockerRuntime` without changes to the pool, proxy, or route handlers.
+
+#### Scenario: Swapping runtimes does not break callers
+- **WHEN** `AppState::comfyui_pool` is constructed with a `Box<dyn ContainerRuntime>` instead of a concrete `DockerRuntime`
+- **THEN** all callers compile and behavior is unchanged for the `DockerRuntime` instance

openspec/changes/comfyui-multi-tenant-api-only/specs/video-agent-per-user/spec.md

@@ -0,0 +1,41 @@
+## ADDED Requirements
+
+### Requirement: video-agent handlers route through the pool
+Every handler under `routes/video_agent.rs` SHALL resolve the caller's `user_id` from the JWT cookie and operate on `state.comfyui_pool.get_or_start(user_id)` instead of the global `state.comfyui` singleton. The wire format of existing endpoints SHALL NOT change.
+
+#### Scenario: status reflects per-user container
+- **WHEN** user 7 calls `GET /api/video-agent/status`
+- **THEN** the response describes user 7's container only, including a new `containerState: "starting" | "ready" | "stopped"` field, and the pre-existing JSON keys are preserved
+
+#### Scenario: start is idempotent per user
+- **WHEN** user 7 calls `POST /api/video-agent/start` twice in a row
+- **THEN** the first call starts (or reuses) the container and returns 200, the second call returns 200 immediately without spawning a duplicate container
+
+#### Scenario: stop only stops the caller's container
+- **WHEN** user 7 calls `POST /api/video-agent/stop`
+- **THEN** user 7's container is stopped and removed, and any other user's running container is unaffected
+
+#### Scenario: chat / analyze / generate-* call the caller's container
+- **WHEN** user 7 calls any of `/api/video-agent/{chat,analyze,character-prompts,generate-character,scene-prompt,generate-scene,generate-tts,full-workflow}`
+- **THEN** the handler forwards the request to user 7's container's HTTP endpoint and returns the response
+
+### Requirement: Backwards-compatible response shape
+Existing JSON keys returned by `/api/video-agent/*` SHALL remain present and identical to the singleton-era response. Only additive fields are allowed.
+
+#### Scenario: status payload is a superset
+- **WHEN** the frontend (unchanged) parses the response of `/api/video-agent/status`
+- **THEN** every field it currently reads is still present with the same type and meaning; only `containerState` (and any new optional fields) are added
+
+### Requirement: Authentication required
+All `/api/video-agent/*` routes SHALL require a valid JWT cookie. Anonymous requests SHALL return HTTP 401 without touching the pool.
+
+#### Scenario: Anonymous request blocked
+- **WHEN** an unauthenticated client calls `POST /api/video-agent/start`
+- **THEN** the response is HTTP 401 and no container is started
+
+### Requirement: Removal of global state.comfyui
+The `comfyui: Arc<ComfyUIProcess>` field on `AppState` SHALL be removed once `comfyui_pool` is the sole code path. No call site SHALL retain a reference to the singleton.
+
+#### Scenario: Codebase contains no singleton refs
+- **WHEN** the source tree is grepped for `state.comfyui` (without the `_pool` suffix)
+- **THEN** there are zero matches outside of historical comments

openspec/changes/comfyui-multi-tenant-api-only/tasks.md

@@ -0,0 +1,69 @@
+## 1. ContainerRuntime trait + DockerRuntime
+
+- [x] 1.1 Add `services/container_runtime.rs` with `ContainerRuntime` trait, `ContainerSpec` (image, name, env, port_map, mounts, cpu, mem, mem_swap, pids_limit, read_only, cap_drop, security_opt), `ContainerHandle` (id, host_port).
+- [x] 1.2 Add `services/docker_runtime.rs` implementing `ContainerRuntime` via `bollard` (or `tokio::process::Command` shelling to `docker` if `bollard` is heavy) — `start`, `stop`, `is_alive`, `list_user_containers`.
+- [ ] 1.3 Unit-test `DockerRuntime` against the local Docker daemon: start/stop/list cycle on a hello-world image; assert resource flags via `inspect`. _(skipped this session — needs a live Docker daemon)_
+
+## 2. ComfyUIPool
+
+- [x] 2.1 Add `services/comfyui_pool.rs` with `ComfyUIPool { runtime: Arc<dyn ContainerRuntime>, containers: RwLock<HashMap<i64, PoolEntry>>, ports: PortAllocator(8200..=8299), max: usize, idle_timeout: Duration }`.
+- [x] 2.2 Implement `get_or_start(user_id) -> Result<ContainerHandle>`: dedupe in-flight starts via per-user `Mutex<Option<oneshot::Receiver>>`; allocate port; build `ContainerSpec` with `image=bosapi-comfyui:multi-tenant`, name `comfyui-user-<id>`, env (`BOOSAPI_BUILTIN_KEY`, `BOOSAPI_BASE`, `BOOSAPI_KEY`), volumes for `/data/comfyui-users/<id>/{input,output,temp,user,custom_nodes_user}`, resource caps from §design Decision 5.
+- [x] 2.3 Readiness probe: poll `GET http://127.0.0.1:<port>/system_stats` until 200 or 30 s timeout; mark state `ready`/`failed`.
+- [x] 2.4 Implement `stop(user_id)`, `state(user_id) -> "starting"|"ready"|"stopped"|"failed"`, `touch(user_id)`.
+- [x] 2.5 Enforce concurrency cap: when `containers.len() >= 30`, return `PoolError::AtCapacity`; map to HTTP 503 `Retry-After: 60` at the proxy/route layer.
+- [x] 2.6 Spawn idle-reaper tokio task: every 60 s, `stop` containers whose `last_used_at` > 60 min AND have no active WS connections.
+- [x] 2.7 Reconcile on startup: `runtime.list_user_containers()` → seed `containers` map with existing `comfyui-user-*` entries.
+
+## 3. AppState wiring
+
+- [x] 3.1 Add `comfyui_pool: Arc<ComfyUIPool>` to `AppState` in `src/lib.rs` (keep `comfyui` field temporarily behind `cfg(feature = "legacy_comfy_singleton")` for rollback).
+- [x] 3.2 Construct the pool in `lib.rs::build_app_state` from `DockerRuntime::new()` and the existing `BuiltinKeys.comfyui_agent` secret.
+- [x] 3.3 Spawn the idle-reaper task at startup; add a graceful-shutdown hook that `stop`s any container we started in this process.
+
+## 4. /comfy/* reverse proxy
+
+- [x] 4.1 Add `routes/comfy_proxy.rs` mounting `any("/comfy/*path")` and `get("/comfy/ws")` (WS upgrade) using `axum::extract::ws` + `reqwest` (HTTP) / `tokio_tungstenite` (WS-to-WS).
+- [x] 4.2 Auth middleware: extract JWT cookie, resolve `user_id`; on missing/invalid, return 401 without touching the pool.
+- [x] 4.3 HTTP forwarder: strip `/comfy` prefix; copy method, path, query, headers (drop hop-by-hop), body; stream response back unchanged.
+- [x] 4.4 WebSocket forwarder: open server-side WS to `ws://127.0.0.1:<port>/ws?...`; pump frames bidirectionally; close on either-side close.
+- [x] 4.5 Cold-start hold: if `pool.state(user_id)` is `starting`, await readiness up to 30 s before forwarding; on timeout return 504.
+- [x] 4.6 Touch `pool` on every HTTP request and every inbound/outbound WS frame.
+- [x] 4.7 Register the router in `lib.rs::build_router` so `/comfy/*` is reachable.
+
+## 5. /api/video-agent/* refactor
+
+- [x] 5.1 In `routes/video_agent.rs`, add a helper `resolve_user_id(headers) -> Result<i64, AppError>` reading the JWT cookie.
+- [x] 5.2 Rewrite `status` to call `pool.state(user_id)` and (if ready) the container's HTTP endpoint; preserve every existing JSON key, add `containerState`.
+- [x] 5.3 Rewrite `start_comfyui` to `pool.get_or_start(user_id)` (idempotent).
+- [x] 5.4 Rewrite `stop_comfyui` and `restart_comfyui` to act on the caller's container only.
+- [x] 5.5 Rewrite `chat`, `analyze`, `character_prompts`, `generate_character`, `scene_prompt`, `generate_scene`, `generate_tts`, `full_workflow` to forward to `http://127.0.0.1:<port>/...` for the caller's container.
+- [x] 5.6 Anonymous-request guard: every handler returns 401 if `resolve_user_id` fails.
+
+## 6. Image build
+
+- [x] 6.1 Add `docker/Dockerfile.comfyui` based on user's existing `bosapi-comfyui:ready` lineage; copy patched `comfy_source/comfy_api_nodes/util/_helpers.py` and `bosapi_comfy_nodes.py` into `/workspace/ComfyUI/`. _(implemented as `docker/Dockerfile.comfyui.multi-tenant` to keep the legacy singleton image untouched until rollout 9.4)_
+- [x] 6.2 ENTRYPOINT script reads `BOOSAPI_BUILTIN_KEY`/`BOOSAPI_BASE`/`BOOSAPI_KEY` and execs `python main.py --listen 0.0.0.0 --port 8188 --comfy-api-base "$BOOSAPI_BASE/api/comfyui-agent"`. _(`docker/comfyui-entrypoint.sh`)_
+- [x] 6.3 Tag image as `bosapi-comfyui:multi-tenant`; update `build-and-run.sh` to build it before launching the API.
+- [ ] 6.4 Smoke-test: `docker run --rm --cpus=2 --memory=2g -e BOOSAPI_BUILTIN_KEY=... -p 18188:8188 bosapi-comfyui:multi-tenant` and curl `/system_stats`.
+
+## 7. Frontend integration
+
+- [ ] 7.1 Update video-agent panel to render full-screen loading overlay when `containerState === "starting"`; poll `/api/video-agent/status` every 2 s.
+- [ ] 7.2 Add a `/comfy` entry point (link or iframe) gated on the user being authenticated; show `containerState` while loading.
+- [ ] 7.3 Handle 503 from `/comfy/*` and `/api/video-agent/*` by showing a "服务繁忙，请稍候" page and retrying after `Retry-After`.
+
+## 8. Verification
+
+- [x] 8.1 `cargo check --all-targets` green.
+- [ ] 8.2 `cargo test` green for new pool / proxy / route tests; pre-existing failures from task #27 explicitly out of scope.
+- [ ] 8.3 Manual two-user test on staging: user A uploads image into ComfyUI, user B starts a fresh container and confirms A's file is NOT visible.
+- [ ] 8.4 Manual cold-start test: stop user A's container, hit `/comfy/`, confirm full-screen loader shows for ~10–20 s, then ComfyUI loads.
+- [ ] 8.5 Manual cap test: start 30 containers, confirm 31st request returns 503 with `Retry-After`.
+- [ ] 8.6 Idle-reap test: leave a container idle for 65 minutes, confirm `docker ps` no longer lists it and the next request cold-starts.
+
+## 9. Rollout
+
+- [x] 9.1 Land 1–6 behind `multi_tenant_comfy` feature flag (default off) so legacy singleton path keeps working.
+- [ ] 9.2 Enable in staging; run §8 manual tests.
+- [ ] 9.3 Enable in production; monitor pool metrics for 1 week.
+- [ ] 9.4 Remove `state.comfyui` singleton field, the feature flag, and any compat branches; archive legacy `docker-compose.comfyui.yml` references.