[CI] Rebaseline gsm8k fully-async E2E regression thresholds

[SumanthRH]

What does this PR do?

Rebaselines the regression thresholds for the nightly gsm8k fully-async E2E CI run. The
loss/avg_final_rewards gate has been failing on every nightly run since 30th June
(22/22 of the last 22 runs), and eval/all/avg_score has been failing intermittently
(3/22). Both thresholds were calibrated on 23rd Feb 2026 (#1199) and were never updated
across three intentional changes to the fully-async recipe.

Why the metrics moved

Pulled every run in the gsm8k_fully_async_ci wandb project (348 runs, back to 23rd April)
and diffed the full summary metric set across the step changes. Four discrete events, each
landing on the first nightly after a specific merge:

regime	loss/avg_final_rewards	eval/all/avg_score	reward/avg_pass_at_5	vllm/prefix_cache_hit_rate	clip_ratio	grad_norm
23 Apr – 17 Jun (original baseline)	0.315	0.588	0.775	0.152	0.0000	0.30
19 Jun – 29 Jun	0.285	0.500	0.745	0.921	0.0000	0.28
30 Jun – 13 Jul	0.238	0.410	0.675	0.921	0.0133	0.163
14 Jul – 19 Jul	0.238	0.500	0.678	0.152	0.0130	0.164
20 Jul – today	0.240	0.575	0.680	0.152	0.0130	0.163

1. [async] add flag to toggle whether to clear kv cache after weight sync for fully async training, update default max_staleness_steps to 0 #1798 (18 Jun) added trainer.fully_async.clear_kv_cache_on_weight_sync=false to the
   recipe, so rollouts began reusing KV computed under stale weights.
   vllm/prefix_cache_hit_rate 0.15 → 0.92 and ttft 1.28s → 0.14s.
2. [chore] Raise assertion error when policy_loss_type uses recomputed logprobs for fully async RL #1850 (30 Jun) set policy_loss_type="rollout_is". This is the dominant cause and it
   has never recovered. Before it, the default regular loss recomputes old logprobs, so at
   update_epochs_per_batch=1 the IS ratio is identically 1 — clip_ratio was exactly
   0.0000 on every run for two months, i.e. there was no importance-sampling correction at
   all despite max_staleness_steps=4. [chore] Raise assertion error when policy_loss_type uses recomputed logprobs for fully async RL #1850 fixed exactly that (issue [fully async] Inform/warn/break about log_probs=old_log_probs #1315) and added an
   assertion forbidding the old behaviour. Correcting it halved grad_norm and lowered
   measured step-9 reward.
3. [generator] Add per global_step cache salt for each trajectory to invalidate super stale KV #1836 (14 Jul) added the per-global_step KV cache salt, reversing [async] add flag to toggle whether to clear kv cache after weight sync for fully async training, update default max_staleness_steps to 0 #1798's stale-KV
   reuse (prefix_cache_hit_rate back to 0.152). eval recovered 0.41 → 0.50.
4. [fix] Log evaluation metrics with the same step in FullyAsyncRayPPOTrainer #1929 (20 Jul) moved metric logging to after eval. Pre-fix, metrics were logged
   before eval ran, so the final-step eval was never logged at all and the summary held
   the step-8 eval; post-fix it holds step 9. eval 0.50 → 0.575. This is a logging
   discontinuity, not a quality change.

So eval/all/avg_score is back at its pre-June level (0.575 vs 0.588) — but only just above
the old 0.56 gate, which is why it now trips intermittently. loss/avg_final_rewards is
legitimately lower because the recipe now applies the off-policy correction it was always
supposed to.

New thresholds

Same methodology as #1199 (5% allowance from min/max of the last 20 runs), applied to the
current post-#1929 regime:

threshold	old	new	last-20 min / max
EVAL_ACC_MIN_VALUE	0.56	0.50	0.5292 / 0.5959
TRAIN_ACC_MIN_VALUE	0.32	0.20	0.2141 / 0.2609
AVG_NUM_TOKENS_MAX_VALUE	283	283 (unchanged)	256.4 / 268.6
LOGPROBS_DIFF_MAX_VALUE	0.040	0.040 (unchanged)	0.0160 / 0.0180

The two passing thresholds are left alone — tightening a gate that never fired isn't part of
fixing the regression and would only add flake risk.

[SumanthRH]

Replay results confirming the attribution

Replayed three variants as Anyscale jobs on l40s_test (g6e.12xlarge, 4x L40S — same GPU
count as l4_ci, so the recipe runs unchanged), logging to a separate
gsm8k_fully_async_ci_triage wandb project. All reached global_step=9 like the nightly.

variant	loss/avg_final_rewards	eval/all/avg_score	pass@5	prefix_cache_hit	clip_ratio	grad_norm
jun17good — 2b31a747 (17 Jun, last green) on its matching ray 2.51.1 image	0.3211	0.6035	0.8047	0.1517	0.0000	0.3082
regularloss — today's main, only policy_loss_type=regular (pre-#1850)	0.2906	0.6588	0.7695	0.1517	0.0000	0.2120
kvclear — today's main + clear_kv_cache_on_weight_sync=true	0.2172	0.5906	0.6289	0.1515	0.0126	0.1540
nightly main for reference (last 5)	0.2141 – 0.2484	0.5292 – 0.5883	0.64 – 0.73	0.1517	~0.013	~0.16

Three things this settles:

1. The 17 Jun commit still reproduces the original baseline: 0.3211 vs 0.3242 measured in CI
   that day. So the drop is entirely attributable to code/config changes — no external drift
   (dataset, vLLM, image) and no L40S-vs-L4 artifact.
2. clip_ratio is exactly 0.0000 for both regular-loss runs and nonzero only with
   rollout_is, directly confirming that before [chore] Raise assertion error when policy_loss_type uses recomputed logprobs for fully async RL #1850 the recipe applied no
   importance-sampling correction at all despite max_staleness_steps=4. grad_norm splits
   the same way (0.21–0.31 for regular, 0.15 for rollout_is).
3. Clearing the KV cache no longer changes anything. kvclear gives prefix_cache_hit
   0.1515 vs unmodified main's 0.1517 and lands inside the nightly reward range, so [generator] Add per global_step cache salt for each trajectory to invalidate super stale KV #1836's
   per-global_step cache salt fully closed the stale-KV hole that [async] add flag to toggle whether to clear kv cache after weight sync for fully async training, update default max_staleness_steps to 0 #1798 opened. The flag is
   effectively inert now — which also makes this run a valid stand-in for an unmodified control.

Measured decomposition of the 0.32 → 0.24 drop:

step	train reward
17 Jun baseline	0.3211
today's main with the pre-#1850 loss	0.2906 (−0.031 from everything else since 17 Jun)
today's main as CI runs it (rollout_is)	~0.24 (−0.05, i.e. the bulk, from #1850)

So #1850 is the dominant cause and it is a deliberate correctness fix, which is what justifies
rebaselining rather than reverting. The residual −0.031 is not stale KV (point 3 above); the
most likely candidate is #1802's fully-async consumer refactor, which is worth a separate look
but doesn't change the threshold decision.

One follow-up still running: rollout_is with the ScaleRL/CISPO band (eps_clip_low=1.0,
eps_clip_high=4.0) instead of the DAPO-tight (0.8, 1.2) it uses today. If that recovers
reward it's a tuning lead for the recipe, tracked separately from this CI fix.

🤖 Generated with Claude Code