feat(server): add proactive channel probes and load-aware stable routing / 服务端新增主动探活与负载感知稳定路由 (#330)
* wip: preserve local changes before upstream merge * Improve proxy routing stability and add stable-first strategy * Sync explicit-group routing strategy to eligible source routes * Fix proxy retry and runtime health review issues * fix: make proxy channel retry attempts configurable * fix: harden proxy retry recovery and bookkeeping * fix: address pr 257 review comments * feat(proxy): add sticky session channel coordination * feat(server): add proactive channel probes and load-aware stable routing * test(server): stabilize session load routing selection assertion * fix(proxy): harden codex forwarding stability * fix(router): rotate stable-first channels by healthy order * fix(server): address probe review feedback * fix(server): address probe review follow-ups * fix(ci): address CodeQL and core test failures * feat(server): refine stable-first routing recovery and observation * Finish PR 330 settings and review fixes * Fix chat site endpoint CI mock * Address latest PR 330 review comments --------- Co-authored-by: Cita <juricek.chen@gmail.com>
This commit is contained in:
@@ -128,6 +128,25 @@ Metapi 的路由引擎按多因子加权选择最优通道。
|
||||
|--------|------|--------|------|
|
||||
| `TOKEN_ROUTER_CACHE_TTL_MS` | Token 路由缓存 TTL(毫秒) | `1500` | >= 100 |
|
||||
|
||||
### 模型弹活探测
|
||||
|
||||
用于异步校正“上游 `/models` 能看到,但实际调用不可用”的假阳性模型。
|
||||
|
||||
| 变量名 | 说明 | 默认值 | 范围 |
|
||||
|--------|------|--------|------|
|
||||
| `MODEL_AVAILABILITY_PROBE_ENABLED` | 启用后台模型可用性探测 | `false` | `true` / `false` |
|
||||
| `MODEL_AVAILABILITY_PROBE_INTERVAL_MS` | 后台定时探测间隔(毫秒) | `1800000` | >= 60000 |
|
||||
| `MODEL_AVAILABILITY_PROBE_TIMEOUT_MS` | 单次探测超时(毫秒) | `15000` | >= 3000 |
|
||||
| `MODEL_AVAILABILITY_PROBE_CONCURRENCY` | 单账号模型探测并发数,默认串行避免对上游形成批量测活 | `1` | 1 ~ 16 |
|
||||
|
||||
说明:
|
||||
|
||||
- 探测任务在后台异步执行,不阻塞前台请求。
|
||||
- 只对看起来像对话/推理模型的条目发最小化探测请求,明显的 embedding、rerank、moderation、tts、image/video 等模型会先跳过。
|
||||
- 只有“明确模型不存在 / 不支持 / 无权限”的错误才会写回不可用;超时、5xx、网络抖动会归为“不确定”,不会误杀现有模型。
|
||||
- 为了避免对上游形成意外的批量测活,默认关闭;只有显式开启后才会按间隔执行,且不会在服务启动后立刻扫全量模型。
|
||||
- 同一批探测任务带去重,上一轮未结束时不会并发堆积;默认并发为 `1`,建议只在确认上游允许主动探测时启用。
|
||||
|
||||
### 路由预设建议
|
||||
|
||||
| 场景 | COST_WEIGHT | BALANCE_WEIGHT | USAGE_WEIGHT | 说明 |
|
||||
|
||||
Reference in New Issue
Block a user