From c1ad46fe48a73121989a50e20be094bc5107939c Mon Sep 17 00:00:00 2001 From: Abhijeet Prasad Date: Wed, 8 Jul 2026 16:03:29 -0400 Subject: [PATCH 1/3] chore: update README for v2 release --- README.md | 158 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 87 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index 8ace966..a86b647 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,55 @@ -# Braintrust eval action +# Braintrust Eval Action -This project enables you to run [Braintrust evals](braintrust.dev) as part of -your CI/CD workflow in Github, using -[Github actions](https://github.com/features/actions). To use this action, -simply include the following step in an action file: +Run [Braintrust evals](https://www.braintrust.dev) in GitHub Actions and post a +live summary comment on the associated pull request. + +## Quick start ```yaml -- name: Run Evals - uses: braintrustdata/eval-action@v1 - with: - api_key: ${{ secrets.BRAINTRUST_API_KEY }} - runtime: node +name: Braintrust evals + +on: + pull_request: + push: + +permissions: + contents: read + pull-requests: write + +jobs: + eval: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - uses: actions/setup-node@v4 + with: + node-version: 22 + + - name: Run evals + uses: braintrustdata/eval-action@v2 + with: + api_key: ${{ secrets.BRAINTRUST_API_KEY }} + runtime: node ``` -You can configure the following variables: - -- `api_key`: Your - [Braintrust API key](https://www.braintrust.dev/app/settings/api-keys). -- `root`: The root directory containing your evals (defaults to `'.'`). The root - directory must have `node`, `python`, or `go` configured. -- `paths`: Specific paths, relative to the root, containing evals you'd like to - run. -- `runtime`: Either `node`, `python`, or `go` -- `package_manager`: Either `npm` or `pnpm` for a `node` runtime, `pip` or `uv` - for a `python` runtime, or `go` for a `go` runtime. You can omit this for Go. -- `use_proxy`: Either `true` or `false`. If set, `OPENAI_BASE_URL` will be set - to `https://braintrustproxy.com/v1`, which will automatically cache repetitive - LLM calls and run your evals faster. Defaults to `true`. -- `terminate_on_failure`: Either `true` or `false`. If set to `true`, the - evaluation process will stop when an error occurs. Defaults to `false`. +> [!IMPORTANT] You must specify `permissions` for the action to leave comments +> on your PR. Without these permissions, you'll see GitHub API errors. + +## Inputs + +| Input | Required | Description | +| ---------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `api_key` | Yes | Your [Braintrust API key](https://www.braintrust.dev/app/settings/api-keys). | +| `runtime` | Yes | The runtime to use: `node`, `python`, or `go`. | +| `root` | No | Root directory containing your evals. Defaults to `.`. | +| `paths` | No | Paths or glob patterns, relative to `root`, containing evals to run. Defaults to `.`. | +| `package_manager` | No | `npm` or `pnpm` for Node; `pip` or `uv` for Python; `go` for Go. Can be omitted for the default package manager. | +| `use_proxy` | No | Set to `true` to use the Braintrust proxy at `https://braintrustproxy.com/v1`, which can cache repetitive LLM calls and speed up evals. Defaults to `true`. | +| `terminate_on_failure` | No | Set to `true` to stop the eval process when an error occurs. Defaults to `false`. Ignored for Go evals. | +| `github_token` | No | GitHub token used to create or update PR comments. Defaults to `${{ github.token }}`. | ## Full example @@ -36,14 +57,15 @@ You can configure the following variables: name: Run pnpm evals on: + pull_request: push: - # Uncomment to run only when files in the 'evals' directory change - # - paths: - # - "evals/**" + # Uncomment to run only when files in the 'evals' directory change. + # paths: + # - "evals/**" permissions: - pull-requests: write contents: read + pull-requests: write jobs: eval: @@ -52,37 +74,33 @@ jobs: steps: - name: Checkout - id: checkout uses: actions/checkout@v4 with: fetch-depth: 0 - name: Setup Node.js - id: setup-node uses: actions/setup-node@v4 with: - node-version: 20 + node-version: 22 - - uses: pnpm/action-setup@v3 + - name: Setup pnpm + uses: pnpm/action-setup@v4 with: - version: 8 + version: 10 - - name: Install Dependencies - id: install - run: pnpm install + - name: Install dependencies + run: pnpm install --frozen-lockfile - - name: Run Evals - uses: braintrustdata/eval-action@v1 + - name: Run evals + uses: braintrustdata/eval-action@v2 with: api_key: ${{ secrets.BRAINTRUST_API_KEY }} runtime: node + package_manager: pnpm root: my_eval_dir ``` -> [!IMPORTANT] You must specify `permissions` for the action to leave comments -> on your PR. Without these permissions, you'll see Github API errors. - -To see examples of fully configured templates, see the `examples` directory: +For more fully configured workflows, see the `examples` directory: - [`node with npm`](examples/node/npm.yml) - [`node with pnpm`](examples/node/pnpm.yml) @@ -90,36 +108,34 @@ To see examples of fully configured templates, see the `examples` directory: - [`python with uv`](examples/python/uv.yml) - [`go`](examples/go/go.yml) -## Go evals - -The Go runtime executes `go run ${paths}`. To include Go eval results in the PR -comment, print each `ExperimentSummary` as one JSON line after calling -`result.Summarize(ctx)`: - -```go -summary, err := result.Summarize(ctx) -if err != nil { - log.Fatal(err) -} -b, err := json.Marshal(summary) -if err != nil { - log.Fatal(err) -} -fmt.Println(string(b)) -``` +## Runtime behavior + +- **Node and Python:** the action runs `braintrust eval --jsonl` from `root` and + collects the emitted experiment summaries. +- **Go:** the action runs `go run ${paths}` from `root`. To include Go eval + results in the PR comment, print each `ExperimentSummary` as one JSON line + after calling `result.Summarize(ctx)`: -## How it works + ```go + summary, err := result.Summarize(ctx) + if err != nil { + log.Fatal(err) + } + b, err := json.Marshal(summary) + if err != nil { + log.Fatal(err) + } + fmt.Println(string(b)) + ``` -For Node and Python, the action runs `braintrust eval`. For Go, the action runs -`go run` on `paths` from `root`. It collects experiment results emitted as JSONL -and posts them as a comment in the PR alongside a link to Braintrust. For -example: +The action creates or updates a single PR comment with a Braintrust link and +result table. For example: -### Example braintrust eval report +### Example Braintrust eval report **[Say Hi Bot (HEAD-1714341466)](https://www.braintrustdata.com/app/braintrustdata.com/p/Say%20Hi%20Bot/experiments/HEAD-1714341466)** -| Score | Average | Improvements | Regressions | -| ----------- | ----------- | -----------: | ----------: | -| Levenshtein | 0.83 (+3pp) | 8 🟢 | 4 🔴 | -| Duration | 1s (0s) | 16 🟢 | 1 🔴 | +| Score | Average | Improvements | Regressions | +| ----------- | ---------- | -----------: | ----------: | +| Levenshtein | 83% (+3pp) | 8 🟢 | 4 🔴 | +| Duration | 1s (0s) | 16 🟢 | 1 🔴 | From 3af6025e3d2aef62640d025ec9710963139971d2 Mon Sep 17 00:00:00 2001 From: Abhijeet Prasad Date: Wed, 8 Jul 2026 17:19:22 -0400 Subject: [PATCH 2/3] Apply suggestion from @AbhiPrasad --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a86b647..6da602d 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,8 @@ jobs: runtime: node ``` -> [!IMPORTANT] You must specify `permissions` for the action to leave comments +> [!IMPORTANT] +> You must specify `permissions` for the action to leave comments > on your PR. Without these permissions, you'll see GitHub API errors. ## Inputs From 3930b273838b47d95bcd6dcaacf3917a0f7ffc67 Mon Sep 17 00:00:00 2001 From: Abhijeet Prasad Date: Wed, 8 Jul 2026 17:22:17 -0400 Subject: [PATCH 3/3] vp check format --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6da602d..78c5777 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,9 @@ jobs: runtime: node ``` -> [!IMPORTANT] -> You must specify `permissions` for the action to leave comments + + +> [!IMPORTANT] You must specify `permissions` for the action to leave comments > on your PR. Without these permissions, you'll see GitHub API errors. ## Inputs