Turn AI agent execution traces into reproducible regression tests, explain behavioral failures, and verify proposed fixes.
trace2test gives tool-using agents a file-based safety net: normalize a known-good trace, generate editable assertions, replay them fully offline, inspect regressions, and carry a Markdown report into CI.
Requires Node.js 20 or newer. To invoke the published CLI without a global install:
npx --yes --package trace2test t2t --help
npx --yes --package trace2test t2t run tests/agentsIf trace2test is installed in a project, its t2t bin also works directly through npm:
npm install --save-dev trace2test
npx t2t runTo run this repository and its deterministic refund demo locally:
git clone https://github.com/felmonon/trace2test.git
cd trace2test
npm install
npm run build
npm run demo:refundThe demo first passes a safe refund trace, then proves that the same suite catches a variant that skips identity verification. No network or LLM is needed for replay.
The featured live support agent is not a scripted mock. GPT-5.6 chose every tool call at runtime through the codex-exec provider while local stubs implemented the tools. With the safe system prompt, the recorded episode called:
lookup_subscription → check_refund_policy → cancel_subscription → escalate_to_human
The operator then weakened one instruction to permit fast cancellation. The real model run changed its behavior to:
lookup_subscription → cancel_subscription → check_refund_policy → escalate_to_human
That is the regression trace committed in this repository. The deterministic suite catches the cancellation before the required policy check without calling an LLM, so anyone can replay the evidence offline:
npx t2t run examples/live-support-agent/tests/agents/live-support.t2t.yaml \
--trace examples/live-support-agent/traces/regression.jsonl \
--no-colorOpen the static judge dashboard to inspect the committed run snapshot.
LLM evaluation harnesses and trace2test answer different questions. Scoring, judge-based, and benchmark tools measure model or agent quality over datasets: “How good are these answers on average?” trace2test turns production traces into deterministic behavioral contracts: “Did this change alter the tool sequence, arguments, outputs, or safety policy that previously worked?”
trace2test also carries a specific failure through deterministic structural diffing, GPT-5.6 diagnosis, and a sandboxed fix --verify loop that must reproduce a passing trace before reporting VERIFIED. Use evals to measure broad quality and trace2test to stop known behavioral regressions in CI. They complement each other.
The commands below run from the repository root and use the included refund agent. Its mock tools are lookup_order, verify_identity, and refund.
Scaffold a commented starter suite and CI workflow, or generate the starter from a known-good trace when an LLM provider is available:
npx t2t init
npx t2t init --trace traces/good.jsonlThe command never overwrites an existing suite or workflow unless --force is supplied.
Normalize generic JSONL or OpenAI Agents SDK JSONL into the canonical trace model:
npx t2t import examples/refund-agent/traces/good.jsonl \
--trace-format generic-jsonl \
--out .trace2test/importsGenerate a proposed .t2t.yaml suite from a known-good trace. Every response is validated against the strict test-spec schema before it is written.
npx t2t generate examples/refund-agent/traces/good.jsonl \
--trace-format generic-jsonl \
--out .trace2test/live/refund.t2t.yamlLLM commands use the OpenAI API when OPENAI_API_KEY is present. Otherwise they fall back to an installed, authenticated codex CLI; if neither is available, the CLI reports clear setup guidance.
Replay the curated suite against the good trace:
npx t2t run examples/refund-agent/tests/agents/refund.t2t.yaml --no-colorThen exercise the deliberately bad trace and write CI-friendly reports:
npx t2t run examples/refund-agent/tests/agents/refund.t2t.yaml \
--trace examples/refund-agent/traces/regression.jsonl \
--json-output .trace2test/live/regression.json \
--junit .trace2test/live/regression.xml \
--markdown .trace2test/live/regression.md \
--no-colorA caught regression exits with status 1; an execution/configuration error exits with status 2. Every run is also persisted under .trace2test/runs/ for the dashboard.
The YAML assertion engine supports exact, ordered, and contiguous tool-call sequences; typed argument and output matchers; and policies such as “require verify_identity before refund” or “forbid refund after failed verification.” Replay-mock stays deterministic and fully offline.
Compute a structural trace diff locally, then ask the configured LLM to identify the behavioral root cause and propose an exact patch:
npx t2t diagnose .trace2test/live/regression.json \
--baseline examples/refund-agent/traces/good.jsonl \
--target examples/refund-agent \
--out .trace2test/live/diagnosis.mdApply the diagnosis patch only inside a sandbox copy, regenerate the refund trace, and rerun the failing suite:
npx t2t fix .trace2test/live/diagnosis.md --verifyThe command prints VERIFIED only when the fresh trace passes. Use --keep-sandbox to inspect the copy, or --command, --target, --spec, and --trace for another agent.
npx t2t ui --port 4173The Vite + React dashboard reads only local artifacts through its localhost server. It shows run history with pass/fail and timing, per-test and per-assertion detail, side-by-side failing-versus-baseline trace diffs, failure clusters by assertion signature, and diagnosis Markdown with a copyable proposed patch. It makes no external network calls.
Export the same dashboard and its current run data as a static site:
npx t2t ui --export docs-siteThe composite action runs the suite, fails the check on a regression, writes a Markdown PR-comment report, and exposes its path as an output. It does not call GitHub APIs directly.
- name: Run agent regression suite
id: trace2test
uses: felmonon/trace2test/action@v1
with:
specs: tests/agents
- name: Upload Markdown regression report
if: always()
uses: actions/upload-artifact@v4
with:
name: trace2test-pr-comment
path: ${{ steps.trace2test.outputs.report-path }}See the complete workflow example, including an optional candidate-trace override.
trace2test is an ESM-only, strict-TypeScript npm workspaces monorepo:
flowchart LR
subgraph Sources[Trace sources]
OA[OpenAI Agents SDK JSONL]
GJ[Generic JSONL]
PROD[Production agent runs]
end
OA --> N[Format normalizers]
GJ --> N
PROD --> N
N --> C[Canonical Trace model]
C --> GEN["t2t generate<br/>GPT-5.6"]
GEN --> YAML[Editable .t2t.yaml specs]
YAML --> R[Deterministic replay runner]
C --> R
R --> REPORTS[Terminal · JSON · JUnit · Markdown]
REPORTS --> CI[GitHub Action / CI]
R --> FAIL[Failing run artifact]
C --> DIFF[Deterministic structural diff]
FAIL --> DIFF
DIFF --> DIAG["t2t diagnose<br/>GPT-5.6 + diff evidence"]
DIAG --> FIX["t2t fix --verify"]
FIX --> BOX[Sandbox copy]
BOX --> FRESH[Fresh agent trace]
FRESH --> R
| Package | Responsibility |
|---|---|
packages/core |
Canonical Zod trace schema, JSONL normalizers, deterministic assertions, YAML specs, replay-mock, structural trace diff, and terminal/JSON/JUnit/Markdown reporters |
packages/cli |
Commander-based t2t commands, run-artifact persistence, diagnosis orchestration, and sandbox verification |
packages/llm |
One LlmClient interface with OpenAI API and local codex exec implementations, strict structured output, validation, and one retry |
packages/ui |
Vite + React dashboard, localhost artifact server, and static snapshot exporter |
examples/refund-agent |
Deterministic tool-using agent, seeded good/regression traces, curated test suite, and demo script |
examples/live-support-agent |
GPT-5.6-driven support agent with local tools and genuine safe/regression episodes recorded through codex-exec |
action |
Composite GitHub Action that produces a portable Markdown regression report |
The deterministic core, replay runner, and dashboard artifact processing never require an LLM or network access. LLM calls remain behind packages/llm.
npm test
npm run typecheck
npm run lint
npm run build
npm run demo:refund
npm run publish:dry-runOn macOS systems that quarantine newly installed native binaries, follow the install workaround in AGENTS.md: install with --ignore-scripts, clear quarantine attributes, rebuild, and clear the attributes again.
trace2test was built during OpenAI Build Week 2026 primarily through OpenAI Codex CLI sessions running GPT-5.6 (model gpt-5.6-sol). The developer directed the work, reviewed the implementation, and live-smoke-tested every milestone.
Read the milestone-by-milestone build log, including the three strict structured-output bugs found only by the real Codex fallback smoke test and how they were fixed.
The product’s generate/diagnose/fix workflow uses GPT-5.6: generate and diagnose call the model through the OpenAI API or the codex CLI fallback, while fix --verify applies the LLM-proposed patch in a sandbox and deterministically proves whether it works.
The main third-party dependencies are Zod for schemas and validation, Commander for the CLI, YAML for test specs, Vite and React for the dashboard, and Vitest for tests.
MIT © 2026 Felmon Fekadu.