Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions docs/mri.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,20 @@ leaderboard = benchmark.leaderboards[0]

# Get the standings
standings = leaderboard.get_standings() # Returns a pandas dataframe
```

Every model evaluation on a leaderboard is carried out by a **job**. The `jobs`
property exposes those jobs so you can drill into the raw comparison results
behind the standings:

# Access the jobs that ran for this leaderboard
```python
# All jobs that have run for this leaderboard (most recent first)
for job in leaderboard.jobs:
results = job.get_results()
print(job.name, job.get_status())
results = job.get_results() # RapidataResults for that evaluation
```

Each entry is a full `RapidataJob`, so the usual methods (`get_results()`,
`get_status()`, `view()`, ...) are available. See
[Model Ranking Insights Advanced](mri_advanced.md#accessing-the-underlying-jobs)
for more.
28 changes: 28 additions & 0 deletions docs/mri_advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,34 @@ participant = benchmark.participants[0]
participant.delete()
```

## Accessing the Underlying Jobs

The standings and win/loss matrix are aggregates. If you need the raw responses
behind them, use the leaderboard's `jobs` property: every model evaluation on a
leaderboard runs as a job, and this returns all of them, most recent first.

```python
leaderboard = benchmark.leaderboards[0]

jobs = leaderboard.jobs # list[RapidataJob], newest first
print(f"{len(jobs)} evaluations have run for this leaderboard")
```

Each item is a full `RapidataJob`, so you can pull the detailed comparison
results, check status, or open a job in the dashboard:

```python
for job in leaderboard.jobs:
if job.get_status() != "Completed":
continue

results = job.get_results() # RapidataResults — the individual matchups
df = results.to_pandas() # or work with the raw dict directly
```

Runs that don't yet have an associated job (for example, an evaluation still
being set up) are skipped, so the list only contains jobs you can act on.

## References
- [RapidataBenchmarkManager](/reference/rapidata/rapidata_client/benchmark/rapidata_benchmark_manager/)
- [RapidataBenchmark](/reference/rapidata/rapidata_client/benchmark/rapidata_benchmark/)
Expand Down
Loading