Skip to content

Add blueprint smoke/performance logging tool#2812

Open
ninad164 wants to merge 3 commits into
dimensionalOS:mainfrom
ninad164:trial-blueprint-perf-logging
Open

Add blueprint smoke/performance logging tool#2812
ninad164 wants to merge 3 commits into
dimensionalOS:mainfrom
ninad164:trial-blueprint-perf-logging

Conversation

@ninad164

@ninad164 ninad164 commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Adds a lightweight dev-only tool for bounded full-blueprint smoke/performance runs.

The tool exercises the real DimOS CLI path in a subprocess, for example:

uv run python -m dimos.robot.tool_blueprint_perf \
  --blueprint unitree-go2 \
  --mode replay \
  --viewer none \
  --duration-seconds 20 \
  --output /tmp/unitree-go2-blueprint-perf.json

@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a dev-only blueprint smoke and performance harness. The main changes are:

  • A new dimos.robot.tool_blueprint_perf subprocess runner.
  • Structured JSON output with runtime metadata, CPU and memory metrics, and log tails.
  • CLI argument validation for bounded runs and log tail sizes.
  • Tests for command construction, output writing, process handling, and invalid inputs.
  • Developer docs showing how to use the harness before deeper profiling.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
dimos/robot/tool_blueprint_perf.py Adds the blueprint smoke/perf runner, argument parsing, process lifecycle handling, metric collection, log tailing, and JSON output writing.
dimos/robot/test_tool_blueprint_perf.py Adds tests for command construction, JSON output, log truncation, startup handling, metric fallback, and invalid arguments.
docs/development/profiling_dimos.md Documents using the new bounded harness before deeper profiling.
docs/development/testing.md Adds the harness as an example dev-only tool command.

Reviews (3): Last reviewed commit: "Validate blueprint perf runtime argument..." | Re-trigger Greptile

Comment thread dimos/robot/tool_blueprint_perf.py Outdated
if mode == "replay":
cmd.extend(["--replay", f"--viewer={viewer}", "run", blueprint])
elif mode == "simulation":
cmd.extend(["--simulation=true", f"--viewer={viewer}", "run", blueprint])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Simulation Backend Becomes True

When --mode simulation is used, this builds the real CLI command with --simulation=true. The DimOS CLI parses simulation as a string backend value, so downstream checks expecting values like mujoco or dimsim receive the literal string true; the smoke run can exercise the wrong mode or fail during blueprint startup instead of testing a supported simulation backend.


ended_at = datetime.now(UTC)
wall_clock_seconds = time.monotonic() - monotonic_start
cpu_user_end, cpu_system_end = _safe_cpu_times(ps_proc)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 CPU Metrics Read After Reap

This reads ps_proc.cpu_times() only after the subprocess has already been terminated or waited. Once proc.wait() reaps the child, psutil can raise NoSuchProcess, so successful bounded runs can write cpu_user_seconds and cpu_system_seconds as null instead of the CPU metrics this tool is meant to capture.

"stderr": stderr_snapshot,
},
}
args.output.write_text(json.dumps(result, indent=2) + "\n")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Nested Output Path Crashes

When callers pass a new nested path such as --output /tmp/blueprint-runs/unitree/result.json, Path.write_text() raises FileNotFoundError because the parent directory is never created. The subprocess run can finish and then fail before writing the promised JSON report.

Suggested change
args.output.write_text(json.dumps(result, indent=2) + "\n")
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(result, indent=2) + "\n")

Comment on lines +129 to +133
"--stdout-tail-lines",
type=int,
default=DEFAULT_STDOUT_TAIL_LINES,
help="How many stdout lines to keep in the JSON tail.",
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Negative Tail Count Crashes

The CLI accepts negative tail counts and passes them into deque(maxlen=...). A run like python -m dimos.robot.tool_blueprint_perf --stdout-tail-lines -1 reaches log collection and raises ValueError, so the tool exits with a traceback and no structured JSON report.

Comment on lines +118 to +139
"--duration-seconds",
type=float,
default=DEFAULT_DURATION_SECONDS,
help="How long to let the command run before terminating it.",
)
parser.add_argument(
"--warmup-seconds",
type=float,
default=DEFAULT_WARMUP_SECONDS,
help="How long the process must stay alive to count startup as successful.",
)
parser.add_argument(
"--output",
type=Path,
default=DEFAULT_OUTPUT,
help="Where to write the structured JSON result.",
)
parser.add_argument(
"--command-timeout-seconds",
type=float,
default=DEFAULT_COMMAND_TIMEOUT_SECONDS,
help="How long to wait after terminate() before escalating to kill().",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Validate runtime floats

The tail-line arguments now reject negative values, but the bounded-run float arguments still accept invalid values. A run with --command-timeout-seconds -1 reaches proc.wait(timeout=-1) and raises before the JSON report is written. A run with --duration-seconds nan never satisfies the duration check, so a long-running blueprint is no longer bounded. A negative --warmup-seconds marks warmup as reached immediately, so an early subprocess failure can be reported as startup success. Please validate these values as finite and in the expected range before building ToolArgs.

@ninad164

ninad164 commented Jul 8, 2026

Copy link
Copy Markdown
Author

Addressed the bot feedback in 1106ca8:

  • changed simulation mode to use a named --simulation-backend instead of --simulation=true
  • preserved CPU metrics from the last valid psutil sample before process cleanup
  • create parent directories for nested JSON output paths
  • validate stdout/stderr tail counts as non-negative argparse inputs
  • added tests for these cases

Validation:
uv run pytest -sv dimos/robot/test_tool_blueprint_perf.py → 12 passed
uv run python -m dimos.robot.tool_blueprint_perf --help → passed

@ninad164

ninad164 commented Jul 8, 2026

Copy link
Copy Markdown
Author

Addressed the remaining runtime argument validation comment.

The tool now validates runtime float inputs before constructing ToolArgs:

  • duration must be finite and > 0
  • warmup must be finite and >= 0
  • command timeout must be finite and >= 0

Invalid values now fail through argparse instead of producing tracebacks or unsafe bounded-run behavior.

Validation:
uv run pytest -sv dimos/robot/test_tool_blueprint_perf.py → 25 passed
uv run python -m dimos.robot.tool_blueprint_perf --help → passed

Note: the focused pytest run still surfaces an existing ResourceWarning from dimos/core/coordination/process_lifecycle.py under warning diagnostics. I did not modify that core lifecycle path in this PR because it is outside the scoped blueprint perf tool change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant