Skip to content

feat(mri): add jobs property to leaderboard#651

Merged
LinoGiger merged 1 commit into
mainfrom
feat(mri)/add-leaderboard-jobs-property
Jul 6, 2026
Merged

feat(mri): add jobs property to leaderboard#651
LinoGiger merged 1 commit into
mainfrom
feat(mri)/add-leaderboard-jobs-property

Conversation

@RapidPoseidon

Copy link
Copy Markdown
Contributor

What

Adds a jobs property to RapidataLeaderboard that returns a list of all Rapidata jobs that have run for the leaderboard.

Why

Each model evaluation on a leaderboard is carried out by a job. Previously there was no way from the SDK to enumerate those jobs (e.g. to pull their individual results). This exposes them directly off the leaderboard object.

How

  • Paginates leaderboard/{id}/runs (100/page, newest first) and collects each run's associated jobId.
  • Resolves each id to a RapidataJob via RapidataJobManager.get_job_by_id, so the returned objects are fully usable (get_results(), get_status(), etc.).
  • Runs without an associated job are skipped.

Docs (docs/mri.md) updated with a short usage snippet. pyright src/rapidata/rapidata_client clean; black formatted.

Usage

leaderboard = benchmark.leaderboards[0]
for job in leaderboard.jobs:
    results = job.get_results()

🔗 Session: https://session-ed5c819e.poseidon.rapidata.internal/

Co-Authored-By: lino <lino@rapidata.ai>
@claude

claude Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review

This PR adds a jobs property to RapidataLeaderboard that paginates leaderboard/{id}/runs and resolves each run's jobId into a full RapidataJob via RapidataJobManager.get_job_by_id. The change is small, self-contained, uses TYPE_CHECKING/lazy imports correctly, and the docs snippet is a nice touch.

Findings

  1. src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py:197 — N+1 API calls, no batching.
    [job_manager.get_job_by_id(job_id) for job_id in job_ids] issues one sequential HTTP request per job, on top of the paginated runs requests (100/page). For a leaderboard with hundreds of runs, accessing .jobs means hundreds of blocking round trips. The generated job_api.jobs_get(...) endpoint already accepts an id filter (AudienceAudienceIdJobsGetJobIdParameter.var_in, a comma-separated "in" filter) — worth checking whether that can be used to fetch all jobs in one (paginated) call instead of one call per id, the same way find_jobs/find_job_definitions already batch via the paged endpoint.

  2. Same line — one bad job reference discards the whole list.
    If any resolved job_id no longer maps to a live job (e.g. deleted/archived between when the run was created and when .jobs is called), get_job_by_id will raise (RapidataError/404) and the whole property call fails — even though the pagination loop may have already successfully collected many other ids. There's no per-item error handling, so one stale reference makes leaderboard.jobs completely unusable until whatever's blocking it is resolved. Worth deciding whether to skip/log a failed lookup instead of failing the entire call, similar to how "runs without an associated job are skipped" is already handled for missing job_ids.

  3. Design nit: jobs is an expensive, uncached @property.
    Every other property on RapidataLeaderboard is a cheap in-memory accessor; operations that hit the backend for computed/aggregated data are modeled as explicit methods (get_standings(), get_win_loss_matrix()). jobs breaks that pattern — it does multi-page pagination plus one HTTP call per job on every access, with no caching, so repeated access (e.g. in a loop, or printed twice for debugging) silently repeats the full expensive fetch. Consider naming it get_jobs() to signal the cost and/or caching the result, consistent with the rest of the class.

Nothing else stood out — pagination loop mirrors the existing pattern in rapidata_benchmark.py, and the docs update is accurate and minimal.

@LinoGiger LinoGiger marked this pull request as ready for review July 6, 2026 14:23
@LinoGiger LinoGiger self-requested a review as a code owner July 6, 2026 14:23
@LinoGiger LinoGiger merged commit 16c08d6 into main Jul 6, 2026
2 checks passed
@LinoGiger LinoGiger deleted the feat(mri)/add-leaderboard-jobs-property branch July 6, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants