feat(serverless): add health, run and status commands (CON-688) by justinwlin · Pull Request #316 · runpod/runpodctl

justinwlin · 2026-07-29T22:54:13Z

Adds serverless health, serverless run and serverless status so the deploy→verify loop stays in the CLI instead of dropping to curl with a hand-pasted API key.

Linear: CON-688

What changed

serverless health <id> — wraps GET /v2/<id>/health, body passed through verbatim.
serverless run <id> --input '{...}' — submits the job, polls until terminal, prints the job payload. --input-file <path> and --input - (stdin) for payloads that fight shell quoting; json is validated locally so a quoting mistake is a local usage_error, not an API round trip.
serverless status <id> <job-id> — the follow-up for --no-wait and for a run that hit its --wait budget.
internal/api.InvokeClient — api.runpod.ai/v2 is a different service from the control plane, so it gets its own client with per-call deadlines; auth, user-agent and error parsing shared so codes stay identical.
Two new error codes (timeout, job_failed), registered in client.go, the README table and AGENTS.md; timeout also applied to model add --wait-for-hash so the code means one thing CLI-wide.

Notable decisions

run uses /run + /status polling, never /runsync (the ticket said /runsync by default). /runsync is not synchronous: it holds the connection ~90s then answers with a still-running job, returns no job id until it answers (a timed-out request strands a billed, unpollable job), and sync- job results expire in 1 minute vs 30 for /run. All three legs source-verified in ai-api. Consequently there is no --async flag — that behavior is the only behavior.
--wait defaults to 5m (measured ~95s cold CPU start; ai-api's own ?wait ceiling is 5m). --no-wait ≡ --wait 0. Per-call timeout is clamped to the remaining budget, with a 1s floor so the last poll can still return an answer.
Exit/stream discipline: a FAILED job exits 1, but the worker's traceback is data — payload on stdout, coded error on stderr. Same on timeout, where the error names the serverless status follow-up.
policy/webhook/s3Config are not exposed (scope cut, follow-up below), but a pasted curl envelope now triggers a warning naming the ignored keys instead of being silently double-wrapped.

Full deviation analysis and review triage (2 rounds, 26 findings: 25 fixed, 1 rejected)

Ten deviations from the ticket as written, each source-verified by an independent reviewer against ai-api (rulings: 7 valid, 2 valid-but-understated, 1 fixed-in-round). Highlights beyond the decisions above:

serverless status is a third command the ticket didn't name — but both non-terminal outcomes print a message referencing it; shipping the messages without the command is a dangling reference back to curl. Ruled load-bearing.
Non-object --input (arrays, scalars, null) rejected locally before any API call; matches ai-api's own 400.
timeout is emitted for plain per-call deadlines too (e.g. health against a stalled host), where nothing is left running; the README now distinguishes the two cases rather than the code being narrowed (narrowing would thread wait-awareness through do() for little gain).
The ticket's "ship health only" smaller cut was declined: health answers "is the endpoint up", not "does my handler work", and run is what the auth-friction problem is about.

Key review fixes: dropping /runsync at the design level (billed unpollable jobs); payload fidelity (64-bit ints survived; handler's gpuTypeId no longer renamed); job id captured once instead of re-read from poll responses (a status-only body used to blank it → GET /status/ → spurious not_found); poll clamping to the wait budget now tested (a mutation had --wait 3s running 26.5s with tests green); retry backoff unified (429 hammering at 2 req/s → shared nextPollInterval); --no-wait on a 200 with no job id exits 1 instead of silently succeeding.

Rejected (1): "--wait may overshoot by up to 1s" — the 1s floor is deliberate; clamping the last poll to a few ms guarantees a timeout instead of an answer. The README/help now state the actual bound.

Testing

Unit: table-driven, httptest for the invoke client, injected mock for the commands — payload fidelity, envelope detection, exit-code mapping, retry policy, deadline clamping in both directions, stdout/stderr discipline. cmd/serverless 74%. Gates: gofmt/go vet clean, go test ./... all 15 packages, docs-gen no diff.

Evidence: stub-host repros, live prod e2e with cleanup proof

Stub invoke host (RUNPOD_INVOKE_URL, free):

# job id survives a poll response that drops it (was: 3 polls to /ep1/status/ )
$ serverless run ep1 --input '{}' --wait 3s
GET /ep1/status/job-d x4 -> {"error":"gave up waiting for job job-d ... poll it with: runpodctl serverless status ep1 job-d","code":"timeout"}

# 429 backoff: 11 requests in 5s -> 6
# --no-wait, submit answers with no job id: exit 1, api_error (was exit 0, silent)
# curl envelope: note names the ignored policy/webhook/s3Config keys
# --input null / unreadable --input-file: local usage_error

Live prod e2e — throwaway CPU endpoint on runpod/mock-worker:latest, deleted via t.Cleanup:

--- PASS: TestE2E_ServerlessInvokeLifecycle (88.95s)
    /run_waits_for_the_result (51.67s)            # cold start, /run + /status polling
    /no-wait_returns_the_job_id_then_status_follows_it
    /input_from_stdin
    /failed_job_exits_1_with_the_payload_on_stdout
    /wait_budget_too_small_is_an_actionable_timeout  # runs the suggested command verbatim
--- PASS: TestE2E_ServerlessHealth / ...NotFound / ...RejectsBadInputLocally

Payload fidelity before/after: "seed": 12345678901234567000 → 12345678901234567890; "gpuId" → "gpuTypeId" preserved. Cleanup verified: serverless list back to the 6 pre-existing endpoints, both e2e templates 404. (Five pre-existing cli_test.go failures are unrelated — stale shared binary + prod data drift.)

Follow-ups

update the runpodctl agent skill — serverless health/run/status, the timeout/job_failed codes (incl. model add --wait-for-hash moving from cli_error to timeout), the "stdout carries the payload even on failure" rule
expose --policy-ttl / --execution-timeout, webhook, s3Config (most needed when raising executionTimeout to match a long --wait)
serverless cancel <job-id> and /stream support
health answered normally for a just-deleted endpoint (invoke-side caching) — don't treat it as an existence check

lukepiette

Approving. The new commands are genuinely additive — no command/flag/alias collisions, no existing help text changed beyond the generated docs index — and the risky decisions check out: polling is deadline-bounded with clamped per-call timeouts (cannot spin forever, pinned by TestBoundedRequestTimeout and the loop-level budget test), /run over /runsync is right given sync-result expiry, credentials and host conventions reuse the existing RUNPOD_INVOKE_URL/config plumbing, stdout/stderr discipline is clean (payload on stdout even for FAILED, progress on stderr), and output.PrintRaw is genuinely needed to avoid the normalizer corrupting handler payloads. Test suite is high-signal; nothing pre-existing weakened.

Comments (none blocking):

One real contract change is hiding under a feature title: model add --wait-for-hash now emits error code timeout where it emitted cli_error. Exit code and message text unchanged, intentional and tested — but scripts branching on the stderr JSON code field break, so it needs a release-note line.
The timeout code is machine-ambiguous. It covers both "wait budget exhausted, job still running, do NOT re-invoke" and "single call timed out, retry is safe," distinguishable only by message prose — which forces exactly the message-parsing the repo's error-object design exists to avoid. A retryable: true/false field or a second code would be cleaner; fine as a follow-up. Related inconsistency: fetchJobStatus returns the raw last error when the budget is consumed by transient failures, where waitForTerminal special-cases it.
Minor: --no-wait --wait 10m is accepted with the wait silently ignored (one Usagef fixes it); resolveJobInput never checks payload size, so an oversized body costs a full upload before the API's 4xx (a local check would match the PR's own fail-locally philosophy); endpoint/job ids are not url.PathEscaped in /status//health paths.
Coordinate land order with #317: both PRs touch the error-code doc block in internal/api/client.go, internal/output/output.go, and both add a /health client (this PR's raw InvokeClient.EndpointHealth vs #317's typed Client.GetEndpointHealth). No compile collision, but whichever lands second should actually consolidate the duplicate, as #317's body already promises.

…rors (CON-688) invoke (api.runpod.ai/v2) is a different service from the control plane, so it gets its own client: per-call context deadlines instead of one client-wide timeout (a /runsync outlives 30s routinely), while reusing the auth header, user-agent and error unwrapping so codes stay uniform. the invoke host is now resolved in one place (invokeBaseURL) shared with the reported endpoint urls. adds two codes to the vocabulary: timeout (the cli stopped waiting; the work may still be running) and job_failed (terminal status other than COMPLETED), plus internal/clierr so a command can return a usage_error the Execute sink understands.

closes the deploy-verify loop without dropping to curl: health wraps GET /health, run invokes with a json payload (--input, --input-file or stdin) and waits for a terminal job status, status follows a job submitted earlier. run defaults to /runsync and keeps polling /status when the api hands back a still-running job, so the default path works for jobs longer than the ~90s the invoke api holds a connection. --async only changes the submit route; --no-wait submits and prints the job id. waiting is bounded by --wait (default 5m) and a bound that runs out is reported as a timeout with the status command to run next, not as a broken endpoint. the job payload always goes to stdout (including a failed worker's error); progress notes and errors stay on stderr.

… regenerate docs (CON-688)

…s out mid-poll (CON-688) found while invoking a live endpoint with --async --wait 8s: the last poll was clamped to the few milliseconds left of the budget, so it always timed out, and that transport timeout was reported instead of the wait timeout — the caller lost the 'poll it with serverless status <ep> <job>' hint on exactly the run that needs it. a poll now keeps a one-second floor, and an exhausted budget always reports the wait timeout with the transient failure demoted to a stderr note. also stop rendering a sub-second budget as "0s".

health against a real endpoint plus a bogus id, and a throwaway cpu endpoint on the public runpod/mock-worker image invoked via runsync, /run and a failing handler. workersMin 0 keeps the endpoint free while idle; the template and endpoint are removed in t.Cleanup.

…g job payloads (CON-688) addresses adversarial review findings on the invoke path. the two structural ones were design bugs, not code bugs: /runsync is not synchronous. ai-api holds the connection for 90s (pkg/api/runsync.go, waitInt := 90000) and then answers with a still-running job, so the cli had to poll /status anyway. two failure modes came free with it: until the response arrives there is no job id, so a request that timed out left a submitted, billed job that could not be polled at all and the error told the caller to re-invoke (double charge); and a sync- job's result is discarded 1 minute after it completes against 30 minutes for /run (pkg/job/job.go), so the "poll it with serverless status" advice was unreliable exactly on that path. `run` now always submits on /run and polls, which costs one extra round trip and makes both impossible. --async is gone: it was only a submit-route switch and the route it switched away from is the one we now always use. job output and the health body are third-party data and were going through output.Print, which rewrites gpuTypeId->gpuId and round-trips through map[string]interface{}. a handler returning an int64 id, a nanosecond timestamp or its own gpuTypeId key got silently corrupted output. added output.PrintRaw (byte-faithful numbers, keys untouched, still sorted; yaml via yaml.Node so it is deterministic too) and api.Job now keeps the response bytes. also: - a 200 with neither id nor status is a bare error object, not a finished job. it used to exit 0 and print `serverless status <ep> ` with an empty job id. - --wait now bounds the submit and status's first fetch too; they ran for the full 30s per-call timeout under --wait 1s, ~30x past the stated bound. - a submit timeout says the job id is unknown rather than pointing at a poll command it cannot supply. - non-object --input is a local usage_error; the api rejects it with a 400 (job.JobInput.Input is a map). - status --wait applies the retry policy to its first call, so one 502 no longer strands the job a timeout error just told an agent to poll. - --wait 0 means "do not poll" on both run and status; --no-wait is that. - model add --wait-for-hash reports the timeout code too, so the code means one thing across the cli.

…meout (CON-688) self-inflicted regression from the previous commit, caught driving the binary: boundedRequestTimeout() clamps a request to what is left of the --wait budget, and a command with no budget (health, --no-wait, a plain status check) has a deadline of "now", so every one of those single calls was being clamped down to the 1s minRequest floor instead of the 30s per-call timeout. a slow but perfectly healthy api call would have timed out. the clamp is now only applied to calls made inside a wait; requestTimeout() is used directly otherwise, and a test pins the deadline a budget-less call gets. also codes the "status but no job id" case api_error instead of letting it fall back to cli_error — an api that answers without an id is not a local mistake. e2e (live prod, throwaway cpu endpoint on runpod/mock-worker, deleted after): the full lifecycle passes end to end, including a cold start completing through /run + /status polling, --no-wait handing back an id that `serverless status` then follows, a FAILED job putting the worker error on stdout while stderr carries job_failed, and a deliberately small --wait producing an actionable timeout whose suggested command works verbatim.

…try policy (CON-688) the poll site's clamp to the wait budget was untested: replacing boundedRequestTimeout(deadline) with requestTimeout() in pollJobStatus left `go test ./...` fully green while a single hung poll ran for the full 30s per-call timeout under --wait 3s (measured 26.5s against a stub invoke host, an 8.8x breach of the bound the readme advertises). the existing tests pinned only the floor and the helper in isolation. adds an upper-bound assertion at the helper and through the wait loop that ships; both kill that mutation. waitForTerminal re-read the job id from every poll response, so a /status body carrying a status but no id blanked it: later polls went to GET /status/ with an empty id and the timeout error named an empty job (`serverless status ep1 `). reproduced against a stub: 1 correct poll then 3 empty-id polls. against prod those 404, and retryablePollError treats 404 as fatal, so a job that was still running would be reported as not_found. the id is now captured once and only status/payload come from a poll response. fetchJobStatus retried on a flat 500ms while the wait loop backed off to 5s, so the first /status call of a --wait command hammered a 429 at ~2 req/s for the whole budget (11 requests in 5s, ~600 on the documented --wait 5m). both loops now share nextPollInterval; the same 429 case is down to 6 requests in 5s.

…g (CON-688) a submit answering with a status but no id exited 0 in silence: HasEnvelope is satisfied by a bare status, so jobOutcome passed it, and the --no-wait branch then skipped its follow-up note because the id was empty — reporting a submitted, billed, unpollable job as a clean success (the exact failure mode dropping /runsync was justified by). the comment there claimed jobOutcome caught it; it did not. now a coded api_error, payload still on stdout. the curl-envelope warning only fired on a lone {"input":...} key, so the common paste {"input":{...},"policy":{"ttl":600000}} was double-wrapped in silence. ai-api reads policy/webhook/s3Config at the *top* level of the request body (pkg/job/job.go:40), so nesting them drops them without a signal. the warning now fires on any top-level "input" key and names the keys that get ignored. --input null slipped past the documented "must be a json object" rule while --input 42 was rejected locally, and an unreadable --input-file was coded cli_error rather than usage_error like every sibling validation in the same function — an agent branching on usage_error got the wrong bucket for a typo'd path.

…ntee (CON-688) the timeout row said the work "may still be running server-side — poll it, do not retry", but invoke.go's do() turns any per-call context deadline into a TimeoutError, so `serverless health` against a stalled host also exits with code "timeout" (reproduced: 30s, empty stdout) with nothing to poll and a retry being the correct move. the control plane reports that same condition as network_error, so one failure gave two contradictory retry instructions. the row now names both cases and how to tell them apart, and drops the claim that network_error is the only retryable code. the readme also promised the per-call timeout "is never allowed to outlast --wait". the deliberate 1s minRequest floor makes that false below one second: --wait 100ms against a hung submit returns after 1052ms. readme and `run --help` now state the actual bound. also records that model add --wait-for-hash timeouts moved from cli_error to timeout, and that HasEnvelope does not cover a submit with a status and no id.

…(CON-688) the vocabulary block is the source the readme table and the agent skill are kept in sync with, so it cannot keep saying "the work may still be running" when a plain per-call deadline (health, a single status check) also emits this code.

… health test (CON-688) Job.MarshalJSON and Job.Field had no production callers — both print sites go through Raw(). the byte-fidelity tests now pin Raw() itself, and the health test that fed the mock a non-json body asserted a state the real client rejects before printing.

…, escape ids (CON-688) review follow-ups: - `--no-wait --wait 10m` was accepted with the wait silently ignored, i.e. the command did the opposite of what the more specific flag asked for. it is now a usage_error, pinned against the real flag set so a rename cannot disable the guard silently. --no-wait on its own is still exactly --wait 0. - resolveJobInput never checked the payload size, so an oversized body cost a full upload before the api's 400 -- minutes on a slow link. the /run limit is 10 MiB (ai-api pkg/api/router.go LimitBodySize(10*MiB)), and the new api.RunBodySize measures what the client actually sends by marshalling the same jobRequest: encoding/json compacts the payload and escapes < > & to six-byte sequences, so len(payload)+envelope is wrong in both directions -- it would refuse a pretty-printed --input-file the api accepts and pass an ampersand-heavy payload it refuses. the check runs before the decode into map[string]interface{}, so an absurd payload is not fully materialised first, and the boundary is asserted against the bytes a real request carries. - endpoint and job ids were interpolated into /run, /status and /health paths unescaped; they are url.PathEscape'd now, as are the invoke urls reported by serverless get/list, which people copy into curl. the invoke service routes on the decoded path, so this is about what goes on the wire, not a claim about what the server then does with it. behavior change to release-note: `model add --wait-for-hash` reports the error code `timeout` where it used to report `cli_error` (exit code and message text unchanged). scripts branching on the stderr json `code` field are affected. still deliberately open, both disclosed in the PR body: - the `timeout` code covers both "the wait budget ran out, the job is still running, do not re-invoke" and "one api call timed out, a retry is safe"; splitting it (or adding `retryable`) changes the error contract, so it is a follow-up rather than a change smuggled into this PR. - `InvokeClient.EndpointHealth` (raw) and CON-689's typed `Client.GetEndpointHealth` hit the same invoke-service endpoint. whichever of the two PRs lands second consolidates them; this one is meant to land last.

…ON-688) CON-689 landed a typed Client.GetEndpointHealth for its --wait poll while this branch added a raw InvokeClient.EndpointHealth for `serverless health`. Both read api.runpod.ai/v2/<id>/health, so the base url, the auth, the id escaping and the error mapping had two homes. As the second of the two to land, this consolidates them, which both PRs said whichever landed second would do. - InvokeClient is now the only client for that service. EndpointHealth returns the raw body, so `serverless health` still prints the api's shape verbatim; the new EndpointHealthCounts decodes it for callers that branch on the numbers. Client.GetEndpointHealth is gone -- the rest control-plane client no longer reaches into the invoke service at all. - waitfor.EndpointHealthGetter takes the poll's context, so the wait's cancellation reaches an in-flight read instead of only being observed between polls. - `serverless create --wait` builds the invoke client only when --wait is set, and wraps each read in the shared "timeout" config key: InvokeClient has no client-wide timeout by design (every call carries its own deadline) and the wait loop's context has none, so an unbounded read would hang the whole wait rather than counting as one failed poll. Pinned by a test that drives it against a server which never answers.

justinwlin force-pushed the justinlin/con-688-serverless-run-health branch from 0acdd87 to d38f28d Compare August 4, 2026 19:44

justinwlin marked this pull request as ready for review August 4, 2026 19:46

lukepiette approved these changes Aug 4, 2026

View reviewed changes

lukepiette mentioned this pull request Aug 4, 2026

feat(pod,serverless): add --wait and --wait-timeout to create (CON-689) #317

Merged

3 tasks

justinwlin force-pushed the justinlin/con-688-serverless-run-health branch 2 times, most recently from 5be2d2f to 890f59e Compare August 5, 2026 18:41

justinwlin added 14 commits August 5, 2026 14:57

docs(serverless): document run/status/health, the new error codes and…

f193c94

… regenerate docs (CON-688)

justinwlin force-pushed the justinlin/con-688-serverless-run-health branch from 890f59e to ad3ee66 Compare August 5, 2026 19:02

justinwlin merged commit c094cac into main Aug 5, 2026
1 check passed

justinwlin deleted the justinlin/con-688-serverless-run-health branch August 5, 2026 19:04

mintlify Bot mentioned this pull request Aug 6, 2026

Update from code changes: serverless health, run, status commands runpod/docs#768

Open

promptless Bot mentioned this pull request Aug 6, 2026

Document serverless run, status, and health CLI commands runpod/docs#770

Open

Navigation Menu

Uh oh!

feat(serverless): add health, run and status commands (CON-688) - #316

feat(serverless): add health, run and status commands (CON-688)#316
justinwlin merged 14 commits into
mainfrom
justinlin/con-688-serverless-run-health

justinwlin commented Jul 29, 2026 •

edited

Loading

Uh oh!

lukepiette left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

justinwlin commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Notable decisions

Testing

Follow-ups

Uh oh!

lukepiette left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

justinwlin commented Jul 29, 2026 •

edited

Loading