Code Translator Agent (Python → JavaScript)

A LangGraph agent that translates Python functions to JavaScript, verifies correctness by actually running the translated code against test cases in a sandbox, and — on failure — feeds the error back to a debugger step for a bounded number of retries.

source code -> [Translate] -> [Verify (sandbox)] --pass--> done
                                     |
                                    fail
                                     v
                                 [Debug] -> [Verify] -> ... (max_attempts)

Why this design

Code translation for small, self-contained functions doesn't need a large multi-agent bureaucracy (planner/manager/understanding agents) — there's no real planning decision being made. What does matter is verification: an LLM can produce plausible-looking but subtly wrong translations (off-by-one in loops, different truthy/falsy semantics between Python and JS, etc.), so the project's core claim is measured, not assumed — every translation is actually executed against test cases, not just eyeballed.

The interesting empirical question this project answers: does a bounded translate → verify → debug loop meaningfully improve correctness over a single-shot translation? That's the pass@1 vs pass@k comparison in the eval harness.

Setup

pip install -r requirements.txt
export GROQ_API_KEY=your_key_here

Requires node and python3 on PATH (used as sandbox runtimes).

Run the eval

python3 eval/run_eval.py --max-attempts 3

This runs every problem in data/problems.json through the pipeline and writes results/eval_results.json with:

• pass@1 — did the first, single-shot translation pass all test cases?
• pass@k — did any attempt (within max_attempts) pass?
• debug_loop_lift — pass@k − pass@1, i.e. how much the verify→debug loop actually bought you
• per-problem attempt history (including captured errors) and latency

Project structure

sandbox/runner.py     # executes python or JS code against test cases in a subprocess, returns pass/fail + errors
agents/translator.py  # LangGraph graph: translate -> verify -> debug loop
eval/run_eval.py       # runs the full problem set, computes pass@1 / pass@k / latency
data/problems.json    # Python source + test cases (Rosetta-Code-style algorithmic problems)
results/               # eval output (generated)

Extending

• More language pairs: sandbox/runner.py already has a RUNNERS dict; add a harness template + subprocess command for the new target language.
• More problems: add entries to data/problems.json — each needs a python source string and a list of {args, expected} test cases. Keep the function name equal to the problem id (enforced by the translator prompt).
• Repo-level translation: this v1 deliberately scopes to single self-contained functions. Multi-file/repo translation is where a Manager (dependency-ordered translation plan) and Understanding (tree-sitter-based dependency graph) agent would actually earn their keep — see project notes for why they're overkill at this scope.