Expose predicted per-Instance queue pressure to least_load

[zhy1658858023]

Background

PR #116 introduced the experimental Proxy-side least_load Instance selection strategy.

PR #118 wired Proxy-maintained per-Instance inflight lifecycle counters into request execution and exposed /debug/instance_loads.

PR #121 extends least_load to use instantaneous per-Instance queue pressure from QueueManager.queue_depth_snapshot(), including:

active_prepare
active_ready
prepare_queue_depth
ready_queue_depth

This is a useful first step, but runtime testing shows that with low RPS, especially rps=1, least_load_score.total can remain 0 most of the time. This is expected for instantaneous counters because queue workers can immediately drain prepare_q / ready_q, and active_prepare / active_ready can be very short-lived.

However, QueueManager already maintains richer predicted pressure state, including:

_instance_pending_tasks
_instance_decode_tasks
_instance_slot_free_ts_s
_instance_prefill_free_ts_s
reservation_seq / prediction timeline
pred_forward_start_ts_ms
pred_first_token_ts_ms
pred_forward_end_ts_ms
pred_worker_free_ts_ms

These internal prediction structures better represent future per-Instance work even after actual Python queues have been drained.

Goal

Expose a safe, Proxy-local per-Instance predicted pressure snapshot from QueueManager, and let least_load use it as an additional scoring input.

The goal is to make least_load_score.total reflect not only instantaneous queue depth, but also predicted unfinished work already assigned to an Instance.

This should make the strategy meaningful even when:

prepare_queue_depth = 0
ready_queue_depth = 0
active_prepare = 0
active_ready = 0

but the Instance still has unfinished predicted prefill/decode work.

Branch hygiene

Follow doc/codex_workflow.md if present.

Start from latest origin/main and use a fresh branch for this issue. Do not continue from previous Codex branches or include previous unmerged PR changes.

Suggested branch name:

codex/issue-123-predicted-instance-pressure

Scope

Expected files may include:

proxy/queue/manager.py
proxy/strategy/least_load.py
proxy/resource/instance_pool.py
proxy/resource/p_control_plane.py
proxy/proxy.py
proxy/README.md

Only modify files needed to expose predicted per-Instance pressure, wire it into least_load, and document/debug it.

Required behavior

1. Add predicted pressure to QueueManager snapshots

Extend QueueManager.queue_depth_snapshot() or add a new method such as:

queue_pressure_snapshot() -> Dict[str, Any]

The snapshot should include the existing instantaneous fields:

prepare_queue_depth
ready_queue_depth
active_prepare
active_ready

and add predicted per-Instance fields derived from reservation/pending/decode state, for example:

pending_prefill_count
active_decode_count
predicted_wait_ms
predicted_prefill_backlog_ms
predicted_decode_backlog_ms
predicted_total_backlog_ms
next_slot_ready_in_ms
prefill_free_in_ms

The exact field names can differ, but they must be stable and documented.

2. Safe derivation rules

Do not expose task payloads or full request content.

Only expose aggregate numeric pressure metrics per Instance.

Suggested derivation:

pending_prefill_count = len(_instance_pending_tasks[instance_id])
active_decode_count = len(_instance_decode_tasks[instance_id])
next_slot_ready_in_ms = max(0, min(_instance_slot_free_ts_s[instance_id]) - now) * 1000
prefill_free_in_ms = max(0, _instance_prefill_free_ts_s[instance_id] - now) * 1000
predicted_total_backlog_ms = max(next_slot_ready_in_ms, prefill_free_in_ms)

If there are stronger existing predicted fields that better represent outstanding work, use them, but keep the snapshot simple and non-invasive.

3. Integrate predicted pressure into least_load scoring

Update LeastLoadStrategy.compute_score(...) to consider predicted pressure when present in the hint.

Potential additional weighted components:

pending_prefill_count
active_decode_count
predicted_total_backlog_ms

Use conservative default weights. Example:

pending_prefill_count_weight: 1.0
active_decode_count_weight: 0.5
predicted_total_backlog_ms_weight: 0.001  # 1000 ms adds 1.0 score

The exact weights can differ, but they must be explained in code comments and README.

The existing score components from PR #121 should remain:

inflight
active_prepare
active_ready
prepare_queue_depth
ready_queue_depth
qps_1m disabled or low weight by default

4. Preserve fallback semantics

If predicted pressure is unavailable, least_load must still work with the existing PR #121 queue-aware score.

If queue/prediction hints are entirely unavailable, fallback to inflight-only scoring.

If all load signals are unknown, fallback to round-robin.

Unknown metrics must not be misrepresented as measured zero.

5. Debug visibility

Update /debug/instance_loads so it exposes the predicted pressure fields and shows how they affect least_load_score.

Debug output should include enough information to explain selection decisions, for example:

{
  "instance_id": "instance_127.0.0.1:9001",
  "inflight": 1,
  "active_prepare": 0,
  "active_ready": 0,
  "prepare_queue_depth": 0,
  "ready_queue_depth": 0,
  "pending_prefill_count": 2,
  "active_decode_count": 1,
  "predicted_total_backlog_ms": 1350,
  "least_load_score": {
    "total": 4.85,
    "components": {
      "inflight": 1,
      "pending_prefill_count": 2,
      "active_decode_count": 1,
      "predicted_total_backlog_ms": 1350
    },
    "weighted_components": {
      "inflight": 1.0,
      "pending_prefill_count": 2.0,
      "active_decode_count": 0.5,
      "predicted_total_backlog_ms": 1.35
    },
    "source": {
      "inflight": "proxy_lifecycle_counter",
      "predicted_total_backlog_ms": "proxy_queue_reservation_timeline"
    }
  }
}

Exact shape can differ, but it must show:

raw predicted pressure fields
weighted score contribution
metric source
weights used

6. Selection-time hint

Ensure the same predicted pressure snapshot used by /debug/instance_loads is also available to least_load during select_instance(...).

Recommended hint shape:

hint = {
    "request": req_obj,
    "queue_depths": queue_mgr.queue_pressure_snapshot(),
}

or keep the existing queue_depths key but extend its content.

Do not change round_robin; it should keep ignoring hints.

7. Documentation

Update proxy/README.md to explain that least_load now considers three layers:

1. Proxy lifecycle pressure: inflight
2. Instantaneous queue pressure: active_prepare / active_ready / prepare_q / ready_q
3. Predicted reservation pressure: pending prefill/decode and predicted backlog time

Document how to inspect the debug output:

curl -sS http://127.0.0.1:8002/debug/instance_loads | python3 -m json.tool

Also document why low RPS may show zero instantaneous queue depth but nonzero predicted pressure during overlapping requests.

Non-goals

Do not implement yet:

kv_aware strategy
KVCache locality scoring
KDN-aware routing
Scheduler Proxy selection based on pool_resource
Scheduler resource-aware routing
IWS decision changes
queue scheduling/release behavior changes
request forwarding behavior changes
Resource Agent sampling changes
pool_resource schema changes

This issue is only about exposing Proxy-local predicted per-Instance pressure and using it inside the existing least_load strategy.

Validation plan

Compile check

PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile \
  proxy/queue/manager.py \
  proxy/strategy/least_load.py \
  proxy/strategy/factory.py \
  proxy/proxy.py \
  proxy/resource/instance_pool.py \
  proxy/resource/p_control_plane.py \
  proxy/queue/*.py \
  test/demo_proxy.py

Unit-style smoke tests

Use fake instances and fake pressure hints to verify:

lower inflight still wins when all other pressure is equal
higher pending_prefill_count loses when inflight is equal
higher predicted_total_backlog_ms loses when instantaneous queue depth is zero
missing predicted fields fall back to PR #121 queue-aware score
missing all queue hints falls back to inflight-only score
all unknown metrics fallback to round-robin
ties are still broken by round-robin

Runtime smoke test

Start Scheduler, Proxy, and at least two Instances.

Start Proxy with:

python3 test/demo_proxy.py --strategy least_load --injection-strategy iws

Observe debug output at high frequency:

while true; do
  date +"%H:%M:%S.%3N"
  curl -sS http://127.0.0.1:8002/debug/instance_loads | python3 -m json.tool
  sleep 0.1
done

Send overlapping requests and verify:

least_load_score.total changes with inflight and predicted backlog
predicted pressure can remain visible even when prepare_q/ready_q are zero
least_load avoids Instances with higher predicted backlog
inflight returns to 0 after requests finish
round_robin still works when selected explicitly

Acceptance criteria

• QueueManager exposes per-Instance predicted pressure without exposing task payloads.
• least_load uses predicted pressure when available.
• /debug/instance_loads displays predicted pressure and weighted score contributions.
• Low-RPS or quickly-drained-queue cases can still show meaningful predicted pressure during overlapping requests.
• Missing predicted pressure does not break existing least_load behavior.
• round_robin remains unchanged.
• No Scheduler routing, KDN, KVCache, IWS, queue release, request forwarding, Resource Agent, or pool_resource schema behavior is changed.