Krrishj18/osmo mission runner

.env

@@ -12,7 +12,7 @@ PROJECT_NAME="airstack"
 # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made 
 # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version.
 # auto-generated from git commit hash
-VERSION="0.19.0-alpha.3"
+VERSION="64594854"
 # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image
 DOCKER_IMAGE_BUILD_MODE="dev"  
 # Where to push and pull images from. Can replace with your docker hub username if using docker hub.

.gitignore

@@ -94,6 +94,11 @@ simulation/ms-airsim/assets/scenes/*
 # Test results
 tests/results/
 
+# OSMO mission results: local runs of osmo/workspace/mission_runner.py write
+# to osmo/results/; `airstack osmo:fetch` downloads to ./osmo-results/.
+osmo/results/*
+osmo-results/*
+
 # Local-only — embedded sibling repo, not part of this branch
 common/rayfronts/
 

docs/tutorials/airstack_on_osmo.md

@@ -547,6 +547,45 @@ osmo workflow cancel $WF
 
 </details>
 
+## Batch missions (unattended runs)
+
+Everything above is the *interactive* workflow. The same pod can instead run
+**missions**: declarative YAML files (in
+[`osmo/missions/`](https://github.com/castacks/AirStack/blob/main/osmo/missions/))
+that script repeated cycles of bring-up → fly → record → tear-down with no
+human attached. Each iteration restarts the containers, records mcap bag
+files (Foxglove's native format — open the `.mcap` directly, no conversion),
+and snapshots container logs and per-step results.
+
+```bash
+# Submit a mission (auto-pins your current branch, like osmo:up):
+./airstack.sh osmo:mission osmo/missions/example_takeoff_land.yaml --pool airstack
+
+# Watch it fly:
+./airstack.sh osmo:logs
+
+# Pull bags + logs + summaries to your laptop — incremental, safe to run
+# mid-mission and again later to top up:
+./airstack.sh osmo:fetch ./results/
+
+# When you have everything (results die with the pod!):
+./airstack.sh osmo:down
+```
+
+A mission step can be any robot task action (`takeoff`, `land`, `navigate`,
+`semantic_search`, `exploration`, `coverage`, …), a timed wait, a topic pub,
+a service call, or an arbitrary `ros2`/shell command. Spec schema and step
+reference:
+[`osmo/missions/README.md`](https://github.com/castacks/AirStack/blob/main/osmo/missions/README.md).
+
+By default the pod **stays alive after the mission ends** so you can
+`osmo:fetch` whenever you're ready (mind the workflow's 24h `exec_timeout`).
+For fire-and-forget batches, submit with `--no-keep-alive`: the pod exits
+cleanly when the mission ends, freeing the GPU — and uploading the results
+directory to object storage automatically if the workflow's `outputs:`
+block is configured (lab-admin setup; see
+[`osmo/README.md`](https://github.com/castacks/AirStack/blob/main/osmo/README.md)).
+
 ## Troubleshooting
 
 | Symptom | Likely cause | Fix |
@@ -572,7 +611,8 @@ osmo workflow cancel $WF
 | Uncommitted edits in the IDE | Pod-local working tree | **No** |
 | `colcon build` outputs (`build/`, `install/`, `log/`) | `/root/AirStack/**/ros_ws/...` | **No** (gitignored Linux x86_64 binaries; rebuild trivially) |
 | Inner-dockerd image cache | Pod-local Docker layer cache | **No** |
-| Bag files, sim recordings, debug screenshots | `/root/AirStack/bags/`, etc. | **No** — pull selectively via `osmo workflow rsync download "$(cat ~/.airstack/osmo-state)" <pod-path>:<local-path>` *before* tearing down |
+| Mission results (mcap bags, logs, summaries) | `/root/AirStack/osmo/results/` | **No** — run `./airstack.sh osmo:fetch` *before* tearing down |
+| Other bag files, sim recordings, debug screenshots | `/root/AirStack/bags/`, etc. | **No** — pull selectively via `osmo workflow rsync download "$(cat ~/.airstack/osmo-state)" <pod-path>:<local-path>` *before* tearing down |
 
 The rule of thumb: **commit + push every time you'd save a file in a
 git-tracked sense.** The Source Control panel is the persistence boundary.

osmo/README.md

@@ -7,11 +7,17 @@ through [NVIDIA OSMO](https://github.com/NVIDIA/OSMO):
 osmo/
 ├── README.md                     # This file (admin / operator reference)
 ├── workflows/
-│   └── airstack-dev.yaml         # The OSMO workflow students submit
+│   ├── airstack-dev.yaml         # Interactive dev workflow (IDE over Remote-SSH)
+│   └── airstack-mission.yaml     # Batch mission workflow (unattended flights)
+├── missions/
+│   ├── README.md                 # Mission spec schema reference
+│   └── example_takeoff_land.yaml # Reference mission: takeoff → hover → land ×3
 └── workspace/
     ├── Dockerfile                # The airstack-osmo-workspace image
     ├── sshd_config               # Pubkey-only sshd config baked into the image
-    └── entrypoint.sh             # Pod startup: sshd, dockerd, clone, airstack up
+    ├── entrypoint.sh             # Pod startup: sshd, dockerd, clone, then
+    │                             #   dev mode (airstack up) or mission mode
+    └── mission_runner.py         # Batch executor (run from the clone, not the image)
 ```
 
 The student-facing walkthrough lives in
@@ -21,9 +27,9 @@ README is the **lab admin / operator** reference: pool requirements,
 workspace image build & push, validation stages, plus a credential summary
 for context.
 
-> **Scope:** developer workflow only. CI/CD on OSMO is **not** part of this
-> integration — the existing `system-tests.yml` + OpenStack orchestrator path
-> is unchanged.
+> **Scope:** developer workflow + batch missions. CI/CD on OSMO is **not**
+> part of this integration — the existing `system-tests.yml` + OpenStack
+> orchestrator path is unchanged.
 
 ## Architecture in one minute
 
@@ -47,6 +53,49 @@ app.foxglove.dev   ── ws ────►  port-forward 8766 ────►
                                             airstack.sh up brings these 3 up
 ```
 
+## Mission mode (batch runs)
+
+`airstack-mission.yaml` reuses the same workspace image and DinD pod, but
+instead of one interactive `airstack up`, the entrypoint hands off to
+[`workspace/mission_runner.py`](workspace/mission_runner.py), which executes
+a declarative mission spec from [`missions/`](missions/) — repeated cycles of:
+
+```
+airstack down → airstack up → wait for PX4 ready → record mcap bags
+→ run steps (takeoff / land / navigate / semantic search / any ros2 command)
+→ collect bags + container logs → airstack down
+```
+
+Submit, monitor, and download:
+
+```bash
+airstack osmo:mission osmo/missions/example_takeoff_land.yaml --pool <gpu-pool>
+airstack osmo:logs                 # follow mission progress
+airstack osmo:fetch ./results/     # rsync bags/logs/summaries to the laptop
+airstack osmo:down                 # cancel (fetch first — results die with the pod)
+```
+
+Key behaviors:
+
+- **The mission spec and runner come from the clone**, not the image — what
+  you push on your branch is what runs. The workspace image only needs a
+  rebuild when `Dockerfile`, `sshd_config`, or `entrypoint.sh` change.
+- **Bags are mcap** (`ros2 bag record -s mcap`) — open the `.mcap` files
+  directly in Foxglove, no conversion or local ROS install.
+- **Results location:** `/osmo/output/airstack-mission-results/<mission>/<stamp>/`
+  with a symlink at `/root/AirStack/osmo/results` (the path `osmo:fetch`
+  pulls). Artifacts are collected even for failed iterations.
+- **`OSMO_MISSION_KEEP_ALIVE`** (default `true`): the pod sleeps after the
+  mission so you can fetch over ssh. Set `false` (or submit with
+  `osmo:mission --no-keep-alive`) for fire-and-forget: the task exits
+  cleanly when the mission ends, freeing the GPU — and if the workflow's
+  `outputs:` block is configured with a destination bucket, OSMO uploads
+  `/osmo/output` automatically on that exit. A **canceled** workflow does
+  not upload outputs, so in keep-alive mode `osmo:fetch` is the retrieval
+  path.
+
+Mission spec schema and step types: [`missions/README.md`](missions/README.md).
+
 ## Pool requirements
 
 The OSMO pool the workflow runs on must satisfy:

osmo/missions/README.md

@@ -0,0 +1,182 @@
+# AirStack mission specs
+
+A **mission** is a declarative YAML file executed by
+[`osmo/workspace/mission_runner.py`](../workspace/mission_runner.py). Each
+mission runs one or more full iterations of:
+
+```
+airstack down → airstack up → wait for PX4 ready → start mcap recording
+→ run steps → stop recording → collect bags + container logs → airstack down
+```
+
+Missions live in this directory so they're versioned with the code they
+exercise — the OSMO pod clones your branch, so whatever you push is what runs.
+
+Submit to OSMO with:
+
+```bash
+airstack osmo:mission osmo/missions/<mission>.yaml --pool <gpu-pool>
+```
+
+Or run locally on any machine that can `airstack up` (no OSMO involved):
+
+```bash
+python3 osmo/workspace/mission_runner.py osmo/missions/<mission>.yaml \
+  --airstack-root "$(pwd)"
+```
+
+`--dry-run` validates the spec and prints the merged config without touching
+Docker.
+
+## Results
+
+```
+osmo/results/<mission-name>/<UTC stamp>/
+├── summary.json              # per-iteration status + durations
+└── iter_001/
+    ├── bags/robot_1/*.mcap   # open directly in Foxglove (no conversion)
+    ├── logs/<container>.log  # docker logs snapshot per container
+    ├── logs/<container>/     # raw /tmp stdout tees (gossip, ddsrouter, relay) + rcl control-stack logs
+    ├── ready.json            # per-robot seconds-to-PX4-ready
+    ├── steps.json            # per-step command, output tail, pass/fail
+    └── iteration.json        # iteration summary
+```
+
+On an OSMO pod the actual storage is `/osmo/output/airstack-mission-results`
+(with `osmo/results` symlinked to it), so a workflow `outputs:` block uploads
+everything automatically when the task exits. Download from your laptop at
+any time while the pod is alive with `airstack osmo:fetch [dest]`.
+
+Artifacts are collected **even when an iteration fails** — a failed flight's
+bag is usually the most interesting one.
+
+## Schema
+
+Top-level keys (everything except `steps` is optional):
+
+| Key | Default | Meaning |
+|---|---|---|
+| `name` | filename stem | Results directory name |
+| `env` | `{}` | Env vars exported before each `airstack up` (`NUM_ROBOTS`, `COMPOSE_PROFILES`, `ISAAC_SIM_SCRIPT_NAME`, …) |
+| `iterations` | `1` | Full up→fly→down cycles. With `environments:` + `environment_order: grouped`, this is **per environment** (total = iterations × #environments) |
+| `iteration_attempts` | `1` | Max times to (re)run a single iteration until it passes. `>1`: a failed/errored iteration is redone (clean down→up→fly→down) up to this many attempts; a `passed`, manually `stopped`, or `abort_mission` outcome is never retried. Failed attempts' artifacts are preserved under `iter_NNN_failed_attempt_K`. With a fixed `SPAWN_SEED` the redo reproduces the same spawn layout |
+| `environment_order` | `round_robin` | With `environments:`: `round_robin` (iteration i → env[(i-1) % n], `iterations` is the total) \| `grouped` (each env runs `iterations` times in a row) |
+| `ready.timeout_s` | `600` | Max seconds to wait for PX4 readiness per iteration |
+| `ready.poll_interval_s` | `5` | Seconds between readiness polls |
+| `record.enabled` | `true` | Record an mcap per robot per iteration |
+| `record.scope` | `gcs` | `gcs` (one mcap on GCS domain 0) \| `robot` (one mcap per robot domain) \| `both` |
+| `record.topics` | tf + odom set | Topics to record; `{robot}` → `robot_N` |
+| `record.all` | `false` | Record **all** topics (`ros2 bag record -a`) — large |
+| `record.exclude` | — | With `all`: regex of topics to drop (`ros2 bag record -a --exclude-regex <regex>`) |
+| `record.required` | `false` | Abort the iteration if any recorder fails to start (don't fly unrecorded) |
+| `on_step_failure` | `abort_iteration` | `continue` \| `abort_iteration` \| `abort_mission` |
+| `up_timeout_s` | `3600` | `airstack up` timeout (first up on a fresh pod pulls images) |
+| `down_timeout_s` | `300` | `airstack down` timeout |
+| `robot_setup_bash` | robot ws `setup.bash` | Workspace sourced before `ros2` commands |
+| `nas_dest` | — | Base path on airlab-storage (e.g. `/volume3/<share>/airstack-missions`). When the `airlab-storage` OSMO credential is set (`airstack osmo:setup`), an OSMO pod rsyncs results to `<nas_dest>/<name>/<stamp>/` then tears itself down. Override per run with `osmo:mission --nas-dest PATH`; suppress with `--no-nas-upload`. Credentials never live in the spec |
+| `steps` | — | Ordered list of steps (below) |
+
+### Steps
+
+Every step type accepts the placeholders `{robot}` → `robot_N` and `{n}` → `N`
+in its strings, and runs once per robot in `robots:` (`all` by default, or a
+list like `[1, 3]`).
+
+**`action`** — send a goal to a robot task action server and wait for the
+result. The step passes when the action result reports `success: true`.
+
+```yaml
+- action:
+    task: takeoff                 # → /robot_N/tasks/<task>
+    goal: {target_altitude_m: 10.0, velocity_m_s: 1.0}
+    timeout_s: 120                # default 120 (per attempt)
+    attempts: 3                   # per-robot retries on failure (default 3)
+    retry_delay_s: 10             # wait between attempts (default 10)
+    feedback_timeout_s: 15        # via gcs: no relay_feedback within this
+                                  # window ⇒ goal presumed lost, retried
+    pass_on_feedback: false       # via gcs: pass as soon as feedback is seen
+                                  # (the task RAN), regardless of success/fail;
+                                  # only "never ran" (no feedback + no result
+                                  # after retries) fails the step
+    robots: all
+    # type: task_msgs/action/TakeoffTask   # derived from task name if omitted
+```
+
+Any step may also carry **`optional: true`** (a sibling key, not inside
+`action`/`run`/…): the step still runs and its result is recorded, but a
+failure neither trips `on_step_failure` nor counts toward an `iteration_attempts`
+redo. Use it for steps whose outcome doesn't gate the iteration — e.g. a `land`
+after the run is already done.
+
+**`pass_on_feedback` + `iteration_attempts` idiom** — to guarantee a task
+*runs* every iteration without caring whether it succeeds: set
+`pass_on_feedback: true` on that action and `iteration_attempts: >1` on the
+mission. If the task never gets feedback (goal lost / never started) after its
+`attempts` retries, the step fails and the whole iteration is redone; once
+feedback is seen the iteration is considered satisfied.
+
+Each robot's goal is logged per attempt and retried independently — a goal
+can be rejected transiently (relay has no GPS fix yet, PX4 position estimate
+not converged, action server still starting), so one robot failing its first
+attempt doesn't fail the step unless it exhausts all attempts.
+
+Available tasks (action type is derived as `task_msgs/action/<CamelCase>Task`):
+`takeoff`, `land`, `fixed_trajectory`, `navigate`, `exploration`, `coverage`,
+`semantic_search`, `chat`. Goal fields are defined in
+[`common/ros_packages/msgs/task_msgs/action/`](../../common/ros_packages/msgs/task_msgs/action/).
+Multi-robot action goals are sent **in parallel** across robots.
+
+**`wait`** — sleep for N seconds (e.g. hover, let a planner run):
+
+```yaml
+- wait: 30
+```
+
+**`run`** — arbitrary command; the escape hatch that makes any ROS 2 command
+work without runner changes. The step fails on non-zero exit unless
+`expect_success: false`. Set `attempts` to retry on failure (e.g. a flaky
+model download) before the step is marked failed.
+
+```yaml
+- run:
+    container: robot_1     # robot_N → exec in the robot container on robot
+                           #   N's DDS domain (ros2 is sourced for you);
+                           # pod → run on the pod itself (cwd = AirStack root);
+                           # any other value → literal container name, domain 0
+    cmd: ros2 topic echo --once /{robot}/odometry
+    timeout_s: 60
+    expect_success: true
+    attempts: 1            # retries on failure (default 1 = no retry)
+    retry_delay_s: 10      # wait between attempts (default 10)
+```
+
+**`topic_pub`** — `ros2 topic pub --once` per robot:
+
+```yaml
+- topic_pub:
+    topic: /{robot}/some_input
+    type: std_msgs/msg/Bool
+    msg: {data: true}
+```
+
+**`service_call`** — `ros2 service call` per robot:
+
+```yaml
+- service_call:
+    service: /{robot}/some_service
+    type: std_srvs/srv/Trigger
+    request: {}
+```
+
+## Notes
+
+- For `NUM_ROBOTS > 1` on Isaac Sim, set
+  `ISAAC_SIM_SCRIPT_NAME: example_multi_px4_pegasus_launch_script.py` in
+  `env:` — the default script spawns a single drone.
+- Missions run unattended: keep `ISAAC_SIM_HEADLESS: "true"`,
+  `ISAAC_SIM_USE_STANDALONE: "true"` and `PLAY_SIM_ON_START: "true"` unless
+  you're watching via the WebRTC livestream profile.
+- Recording camera/LiDAR topics (or `record.all: true`) is the bag-size
+  driver — budget pod `storage:` accordingly.
+- `/tf` and `/tf_static` are in the default topic set because without them a
+  Foxglove 3D panel can't pose anything during replay.