[NV] llm-d-vllm: Add llm-d to the InferenceX benchmarking framework

[ezrasilvera]

Adding llm-d into InferenceX Starting with DeepSeek-V4-Pro FP4 GB200 disaggregated benchmarks (mid-curve + high-tpt)

Summary

Adds llm-d (vLLM) as a new benchmark framework in InferenceX, with a no-k8s multi-node path (Envoy + EPP + P/D sidecar over SLURM/pyxis), and theDeepSeek-V4-Pro FP4 GB200 P/D-disaggregated recipes:

• dsv4-fp4-gb200-llm-d-vllm-mid-curve-megamoe - 1P1D DEP8, conc 256/512/1024
• dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe - 2P1D DEP8, conc 4096

Both keys share one engine/EPP recipe (benchmarks/multi_node/llm-d-recipes/dsv4-fp4-gb200-mid-curve-megamoe.yaml);
topology and concurrency come from the master-config keys, not the recipe.

Notable

• These keys mirror dsv4-fp4-gb200-dynamo-vllm - same SKU, model, ISL/OSL, disagg TP1/DP8/EP8 layout, and the same concurrency points (256/512/1024/4096).
• Emits the same agg_*.json rows through process_result.py (framework=llm-d-vllm, X=conc, Y=tput_per_gpu) and the same _gpus_/_ctx_/_gen_ result-filename convention the multinode ingest already parses. The only downstream addition is
  the framework label in InferenceX-app (separate PR).
• Every reported point is a full 100%-completion sweep: (e.g. 12288/12288 at conc4096).
• One recipe, two topologies: Mid-curve (1P1D) and high-tpt (2P1D) share a single engine/EPP recipe; node counts and concurrency live in the master-config keys - so adding a curve point is a config-key change, not a new recipe file.

Scope

• GB200 DeepSeek-V4-Pro FP4 only. No changes to other SKUs/models/frameworks.

Known merge-blocker

• The image: on both keys is a personal build (ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23). It should be republished to the org registry and the image: lines updated before merge.

Related

• Dashboard label PR (separate repo): registers llm-d-vllm -> "llm-d vLLM" in SemiAnalysisAI/InferenceX-app (FW_REGISTRY) so the new series is recognized and labeled on inferencex.semianalysis.com. https://github.com/SemiAnalysisAI/InferenceX-app/pull/524

Before merge

• Republish image to org registry; update image: on both keys.
• Canonicalize perf-changelog.yaml pr-link from pull/XXX to this PR's number.
• Full sweep green.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-enabled or full-sweep-fail-fast label to this PR — the benchmark sweep only runs on labeled PRs.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-enabled 或 full-sweep-fail-fast 标签 — 基准测试 sweep 仅在带有标签的 PR 上运行。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-enabled or full-sweep-fail-fast label to this PR — the benchmark sweep only runs on labeled PRs.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-enabled 或 full-sweep-fail-fast 标签 — 基准测试 sweep 仅在带有标签的 PR 上运行。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-enabled or full-sweep-fail-fast label to this PR — the benchmark sweep only runs on labeled PRs.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-enabled 或 full-sweep-fail-fast 标签 — 基准测试 sweep 仅在带有标签的 PR 上运行。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🟡 The comment on lines 10-12 says "This is currently not used as we build an use an image that contains all binaries," but benchmarks/multi_node/llm-d/job.slurm (lines 236-241 of the pyxis branch) explicitly sources this file and uses LLMD_BIN_DIR to conditionally bind-mount epp/pd-sidecar/envoy — so the file IS wired into the runtime as an optional override. Also a typo: "build an use" → "build and use". Nit — no runtime impact, but the stale comment will mislead maintainers into thinking they can delete the file.

Extended reasoning...

What the comment claims vs. what the code does

Lines 10-12 of benchmarks/llm-d/binaries.env state:

This is currently not used as we build an use an image that contains all binaries

However, the pyxis branch of benchmarks/multi_node/llm-d/job.slurm explicitly sources this file and consumes LLMD_BIN_DIR to build optional bind-mounts:

[[ -f "${DI_REPO_DIR}/benchmarks/llm-d/binaries.env" ]] && \
    source "${DI_REPO_DIR}/benchmarks/llm-d/binaries.env"
for _bin in epp pd-sidecar envoy; do
    if [[ -n "${LLMD_BIN_DIR:-}" && -x "${LLMD_BIN_DIR}/${_bin}" ]]; then
        PYXIS_MOUNTS+=",${LLMD_BIN_DIR}/${_bin}:/usr/local/bin/${_bin}:ro"
        ...
    fi
done

So binaries.env is wired into the runtime — it acts as the single source of truth for LLMD_BIN_DIR (defaulted to /mnt/lustre01/users-public/sa-shared/llm-d-bins), which the pyxis path uses to swap in mounted binaries and skip a combined-image rebuild.

Step-by-step proof

1. Author enables the pyxis path (the default on GB200 via LLMD_CONTAINER_ENGINE=pyxis set in runners/launch_gb200-nv.sh).
2. job.slurm reaches the pyxis branch and sources binaries.env, populating LLMD_BIN_DIR.
3. If a user has run extract-binaries.sh (which also sources this file), the three binaries exist under $LLMD_BIN_DIR and get bind-mounted into every container.
4. This is the entire point of the mounted-binary path documented in extract-binaries.sh and the Dockerfile — running a stock vllm/vllm-openai image without rebuilding the combined image.

The comment therefore contradicts both the sibling files (extract-binaries.sh, Dockerfile, envoy.yaml docstrings) that describe binaries.env as the single source of truth, and job.slurm itself.

Addressing the counter-argument

One could argue the comment is "arguably accurate for the DEFAULT deployment path" — in the current recipe the combined image bakes binaries in, so the -x check makes the mount loop a no-op unless the operator has explicitly run extract-binaries.sh. That's a fair characterization of when the file has effect, but "currently not used" is stronger than "currently a no-op by default" and it flatly contradicts the sourcing on job.slurm:236. A future maintainer trying to trim "unused" scaffolding would delete this file and quietly break the mounted-binary path.

Impact and fix

No runtime impact — the code works today. The concern is purely documentation drift in a new file this PR is introducing. Fix in one edit:

# This file is sourced by benchmarks/multi_node/llm-d/job.slurm to
# optionally bind-mount extracted binaries in place of the combined
# image's baked-in copies (see extract-binaries.sh). It is a no-op
# by default (the -x check fails when the binaries have not been
# extracted); the Dockerfile also reads the same URLs when building
# and using an image that contains all binaries.

Also fix the "build an use" → "build and use" typo while you're in there.

[claude]

🟡 The README section "Why one entry per leader (not per node)" (lines 80-89) plus the example endpoints.yaml at lines 57-69 both document per-leader emission, but server.sh add_role() (lines 334-343) actually emits ONE endpoint per node — its own comment says "ONE endpoint per node at base_port (hybrid LB: ...)". The lines 91-95 "future work" section describing per-node emission as hypothetical has already been implemented. Docs-only staleness (per-node is the intended behavior — see server.sh:289-291 and its consistency with the multi-node coordinator health poll in lines 519-535), but future maintainers may misdiagnose whether current behavior is intentional. Please update the README to reflect actual per-node emission, or add a note explaining why the leader-only text was preserved.

Extended reasoning...

What the bug is

The new file benchmarks/multi_node/llm-d/README.md (added by this PR) describes the endpoints.yaml file-discovery contract as emitting one entry per leader. Concretely:

• Line 55: "…writes /tmp/endpoints.yaml inside the container with one entry per leader"
• Lines 57-69: the example YAML shows only prefill-0 (PREFILL_LEADER_IP) and decode-0 (DECODE_LEADER_IP)
• Lines 80-89: an entire section titled "Why one entry per leader (not per node)" justifies leader-only emission and warns: "Adding an entry per worker would cause EPP to route directly to a worker, bypassing the engine's internal load balancing"
• Lines 91-95: describes per-node emission as a hypothetical future extension ("we can extend the loop … to emit one entry per IPS[i]")

But the code at benchmarks/multi_node/llm-d/server.sh:334-348 (the add_role() block that writes endpoints.yaml) does exactly the opposite:

def add_role(role, ips, base_port):
    # ONE endpoint per node at base_port (hybrid LB: the node's api-server /
    # sidecar internally load-balances its local DP ranks).
    for i, ip in enumerate(ips):
        endpoints.append({'name': f'{role}-{i}', ...})

add_role('prefill', prefill_ips, VLLM_PORT)     # prefill_ips = all_ips[:pn]
add_role('decode', decode_ips, SIDECAR_PORT)    # decode_ips = all_ips[pn:pn + dn]

The inline comment at server.sh:339-340 literally says the opposite of the README section header: "ONE endpoint per node at base_port".

Step-by-step proof

Take the new high-tpt 2P+1D key (dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe, configs/nvidia-master.yaml). Topology: PREFILL_NODES=4, DECODE_NODES=2, PREFILL_WORKERS=2.

1. submit.sh sbatches 6 nodes. job.slurm collects their IPs in rank order into ALL_IPS = [P0,P1,P2,P3,D0,D1].
2. On the decode leader, server.sh runs the coordinator Python block:
   • prefill_ips = all_ips[:4] = [P0, P1, P2, P3]
   • decode_ips = all_ips[4:6] = [D0, D1]
3. add_role('prefill', prefill_ips, 8200) iterates 4 times → emits prefill-0..prefill-3.
4. add_role('decode', decode_ips, 8000) iterates 2 times → emits decode-0..decode-1.
5. Final endpoints.yaml has 6 entries — not the 2 (prefill-0 + decode-0) the README example shows.

For the 1P+1D mid-curve key it emits 1 prefill + 2 decode entries (still per-node, not per-leader).

Why this is intentional (i.e. the code is right, docs are stale)

Several signals confirm per-node emission is the deliberate design, not a bug:

1. server.sh:289-291 comment explicitly justifies it: "Each decode node runs its own sidecar (SIDECAR_PORT -> local decode vLLM), and endpoints.yaml lists one decode endpoint per node so EPP fans out across all decode ranks."
2. server.sh:519-535 (the decode coordinator's cross-node prefill health poll) iterates over _prefill_ips=( "${_ALL_IPS[@]:0:${PREFILL_NODES}}" ) — polling every prefill node, not just the leader. That poll is only meaningful if all prefill nodes are individually addressable endpoints.
3. --api-server-count is set on every EP-enabled node (server.sh:236-240), plus --data-parallel-hybrid-lb — meaning each node has its own api-server binding, so per-node endpoints are actually the correct interpretation of hybrid-lb.
4. PREFILL_WORKERS=2 in the high-tpt key requires per-node emission: with 2 independent DP engines across 4 prefill nodes, per-leader emission would only expose one of the two engines.

Impact

Zero runtime impact — code is self-consistent and produces the intended routing behavior. But a future maintainer reading the README's "Why one entry per leader (not per node)" section will:

• Believe the current behavior is per-leader when it isn't.
• Reference the warning about "EPP would bypass internal load balancing" and (incorrectly) conclude the current implementation is broken.
• Not realize the "future work" paragraph at lines 91-95 has already been implemented.

Suggested fix

Update benchmarks/multi_node/llm-d/README.md:

• Line 55: change "one entry per leader" → "one entry per node".
• Lines 57-69: expand the example YAML to show all prefill and decode nodes (e.g. prefill-0..prefill-N-1, decode-0..decode-M-1).
• Lines 80-89: rewrite the "Why one entry per leader" section to explain why per-node is used (hybrid-lb + independent api-servers + PREFILL_WORKERS multi-engine).
• Lines 91-95: delete or rewrite the "future work" paragraph since per-node emission is what the code already does. The leader-only mode could be described here as the hypothetical alternative if desired.

[ezrasilvera]

Fixed in d7e4180 - README now documents one endpoint per node (not per leader) to match server.sh add_role(), with corrected example ports (prefill 8200 / decode 8000) and a rewritten rationale section.

[claude]

🟡 Both new keys (mid-curve line 9080, high-tpt line 9125) pin image: ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23, a personal ghcr namespace rather than the SemiAnalysisAI org registry. The PR description's "Before merge" checklist and the in-file comment at 9075-9077 both flag this — surfacing here as a merge-time reminder so it isn't missed. Republish to ghcr.io/semianalysisai/... and update both image: lines before merge.

Extended reasoning...

What the bug is

Both new master-config keys added by this PR pin their engine image to a personal ghcr namespace:

• configs/nvidia-master.yaml:9080 (dsv4-fp4-gb200-llm-d-vllm-mid-curve-megamoe): image: ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23
• configs/nvidia-master.yaml:9125 (dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe): image: ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23

The PR author has explicitly flagged this in two places already: the PR description's "Known merge-blocker" section ("The image: on both keys is a personal build... It should be republished to the org registry and the image: lines updated before merge") and an in-file comment at lines 9075-9077 ("PR follow-up (merge-blocking): the image: on both llm-d-vllm keys below (mid-curve + high-tpt) is a personal build. Republish it to the org registry (ghcr.io/semianalysisai/...) and update these image: lines before merge").

Why it matters

Reproducibility. On the GB200 pyxis path, runners/launch_gb200-nv.sh derives SQUASH_FILE="${SQUASH_DIR}/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" — so the enroot-import cache key is a direct function of the image string. Any historical rerun (e.g., re-ingesting a failed sweep, reproducing a curve point months later) resolves the image via that exact URL. A personal ghcr namespace can be retagged, deleted, or made private without notice by the individual owner, silently breaking reruns.

Step-by-step proof

1. configs/nvidia-master.yaml:9080 sets image: ghcr.io/ezrasilvera/llm-d-nokube-vllm:vllm0.23 on the mid-curve key.
2. configs/nvidia-master.yaml:9125 sets the same image on the high-tpt key.
3. runners/launch_gb200-nv.sh (the llm-d-vllm branch, top of file) reads $IMAGE and constructs ENROOT_URL="docker://ghcr.io#ezrasilvera/llm-d-nokube-vllm:vllm0.23" for enroot import.
4. enroot import authenticates and pulls against ezrasilvera's personal ghcr namespace on every fresh cluster / cache miss.
5. If ezrasilvera retags/deletes/private-marks this tag, step 4 fails; the sweep can no longer be reproduced.

Addressing the refutation

One verifier argued this duplicates the author's own annotations (PR description merge-blocker + in-file comment) and should be dropped as "no new information." That's a fair point, and it's why I'm filing this as nit rather than normal — I'm not claiming the author is unaware. But surfacing it in the code-review pass provides a merge-time reminder against the actual diff (as opposed to a checklist item in the description that can be overlooked when the reviewer clicks Merge). If this shows up alongside other findings in the same review, it's cheap to satisfy; if the maintainer knows the image swap is already in flight, the reminder can be dismissed in a second.

Fix

Republish the combined image (built from benchmarks/llm-d/Dockerfile) to ghcr.io/semianalysisai/ and update both image: lines. The enroot cache key changes with the image string, so this is a valid config change to land in this PR rather than a follow-up — landing it here keeps the initial ingest and all historical reruns pinned to the org-owned image.

[claude]

🟡 The new perf-changelog entry has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (literal XXX placeholder at perf-changelog.yaml:4444). Every other entry in this file resolves to a real PR number — this one alone will 404 if merged as-is. Fix by replacing XXX with 2050 before merge. (The author already listed this under 'Before merge' in the PR description, so filing as a friendly nit rather than a blocker.)

Extended reasoning...

What the bug is. The perf-changelog entry appended by this PR (perf-changelog.yaml:4441-4444) carries a literal pull/XXX placeholder rather than the real PR number:

- config-keys:
    - dsv4-fp4-gb200-llm-d-vllm-mid-curve-megamoe
    - dsv4-fp4-gb200-llm-d-vllm-high-tpt-megamoe
  description:
    - "Add DeepSeek-V4-Pro FP4 GB200 llm-d-vllm disaggregated multinode benchmarks"
    - "Mid-curve 1P+1D DEP8 (conc 256/512/1024)"
    - "High-tpt 2P+1D DEP8 (conc 4096)"
  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

Why the current diff is inconsistent. Every other entry in perf-changelog.yaml resolves to a real PR (the immediately preceding entry ends with pull/2001; earlier entries reference pull/1990, pull/1978, etc.). The pr-link contract of this file is to trace which PR introduced which benchmark set — a literal XXX breaks that contract.

Impact if merged as-is. Clicking the link in the changelog will hit https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, which resolves to no PR (GitHub returns 404 for numeric paths that don't exist, or worse silently redirects). No functionality is affected — the sweep still runs, the config keys still work — but the diagnostic "which PR introduced this row?" trail is broken for the two new dsv4-fp4-gb200-llm-d-vllm keys.

Why existing checks don't catch it. There is no linter validating perf-changelog pr-link values against real PR numbers, so nothing stops the placeholder from shipping. The PR description's 'Before merge' checklist explicitly lists this: 'Canonicalize perf-changelog.yaml pr-link from pull/XXX to this PR's number' — the author has already flagged it as a known merge-blocker to canonicalize.

Step-by-step proof.

1. Open perf-changelog.yaml at line 4444 → observe pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.
2. grep -c 'pull/XXX' perf-changelog.yaml → 1 (only this new entry).
3. grep -oE 'pull/[0-9]+' perf-changelog.yaml | tail -3 → all resolve to real numeric PRs (2001, 1990, 1978).
4. Attempt https://github.com/SemiAnalysisAI/InferenceX/pull/XXX in a browser → GitHub 404.
5. Expected value: pull/2050 (this PR's number, per the PR metadata).

Fix. One-character-scope edit: replace XXX with 2050 on perf-changelog.yaml:4444. No other files need to change.

[ezrasilvera]

Fixed in d7e4180 - pr-link canonicalized from pull/XXX to pull/2050.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28702992096
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28702992096

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28703886323
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28703886323

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28707148515
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28707148515

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28714892207
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28714892207