Add blueprint smoke/performance logging tool#2812
Conversation
Greptile SummaryThis PR adds a dev-only blueprint smoke and performance harness. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (3): Last reviewed commit: "Validate blueprint perf runtime argument..." | Re-trigger Greptile |
| if mode == "replay": | ||
| cmd.extend(["--replay", f"--viewer={viewer}", "run", blueprint]) | ||
| elif mode == "simulation": | ||
| cmd.extend(["--simulation=true", f"--viewer={viewer}", "run", blueprint]) |
There was a problem hiding this comment.
Simulation Backend Becomes True
When --mode simulation is used, this builds the real CLI command with --simulation=true. The DimOS CLI parses simulation as a string backend value, so downstream checks expecting values like mujoco or dimsim receive the literal string true; the smoke run can exercise the wrong mode or fail during blueprint startup instead of testing a supported simulation backend.
|
|
||
| ended_at = datetime.now(UTC) | ||
| wall_clock_seconds = time.monotonic() - monotonic_start | ||
| cpu_user_end, cpu_system_end = _safe_cpu_times(ps_proc) |
There was a problem hiding this comment.
This reads ps_proc.cpu_times() only after the subprocess has already been terminated or waited. Once proc.wait() reaps the child, psutil can raise NoSuchProcess, so successful bounded runs can write cpu_user_seconds and cpu_system_seconds as null instead of the CPU metrics this tool is meant to capture.
| "stderr": stderr_snapshot, | ||
| }, | ||
| } | ||
| args.output.write_text(json.dumps(result, indent=2) + "\n") |
There was a problem hiding this comment.
When callers pass a new nested path such as --output /tmp/blueprint-runs/unitree/result.json, Path.write_text() raises FileNotFoundError because the parent directory is never created. The subprocess run can finish and then fail before writing the promised JSON report.
| args.output.write_text(json.dumps(result, indent=2) + "\n") | |
| args.output.parent.mkdir(parents=True, exist_ok=True) | |
| args.output.write_text(json.dumps(result, indent=2) + "\n") |
| "--stdout-tail-lines", | ||
| type=int, | ||
| default=DEFAULT_STDOUT_TAIL_LINES, | ||
| help="How many stdout lines to keep in the JSON tail.", | ||
| ) |
There was a problem hiding this comment.
| "--duration-seconds", | ||
| type=float, | ||
| default=DEFAULT_DURATION_SECONDS, | ||
| help="How long to let the command run before terminating it.", | ||
| ) | ||
| parser.add_argument( | ||
| "--warmup-seconds", | ||
| type=float, | ||
| default=DEFAULT_WARMUP_SECONDS, | ||
| help="How long the process must stay alive to count startup as successful.", | ||
| ) | ||
| parser.add_argument( | ||
| "--output", | ||
| type=Path, | ||
| default=DEFAULT_OUTPUT, | ||
| help="Where to write the structured JSON result.", | ||
| ) | ||
| parser.add_argument( | ||
| "--command-timeout-seconds", | ||
| type=float, | ||
| default=DEFAULT_COMMAND_TIMEOUT_SECONDS, | ||
| help="How long to wait after terminate() before escalating to kill().", |
There was a problem hiding this comment.
The tail-line arguments now reject negative values, but the bounded-run float arguments still accept invalid values. A run with --command-timeout-seconds -1 reaches proc.wait(timeout=-1) and raises before the JSON report is written. A run with --duration-seconds nan never satisfies the duration check, so a long-running blueprint is no longer bounded. A negative --warmup-seconds marks warmup as reached immediately, so an early subprocess failure can be reported as startup success. Please validate these values as finite and in the expected range before building ToolArgs.
|
Addressed the bot feedback in 1106ca8:
Validation: |
|
Addressed the remaining runtime argument validation comment. The tool now validates runtime float inputs before constructing ToolArgs:
Invalid values now fail through argparse instead of producing tracebacks or unsafe bounded-run behavior. Validation: Note: the focused pytest run still surfaces an existing ResourceWarning from dimos/core/coordination/process_lifecycle.py under warning diagnostics. I did not modify that core lifecycle path in this PR because it is outside the scoped blueprint perf tool change. |
Summary
Adds a lightweight dev-only tool for bounded full-blueprint smoke/performance runs.
The tool exercises the real DimOS CLI path in a subprocess, for example: