feat(hosted-teleop): Go2 teleoperation over Cloudflare/LiveKit broker#2798
Draft
ruthwikdasyam wants to merge 209 commits into
Draft
feat(hosted-teleop): Go2 teleoperation over Cloudflare/LiveKit broker#2798ruthwikdasyam wants to merge 209 commits into
ruthwikdasyam wants to merge 209 commits into
Conversation
…e SFU Implements a new pubsub transport backed by WebRTC DataChannels over Cloudflare's Realtime SFU. Two new classes in dimos/protocol/pubsub/impl/webrtcpubsub.py: - CloudflareSession: manages the WebRTC PeerConnection lifecycle. Opens two CF sessions (publisher + subscriber) so a single process can do loopback pubsub. Runs aiortc on a dedicated background asyncio thread with its own ThreadPoolExecutor (so we don't leak asyncio_N worker threads). Uses negotiated=True placeholder DCs with id=100 during transport establishment to avoid stream-id collisions with CF-assigned ids. - WebRTCPubSub: bytes-on-the-wire pubsub facade matching the LCMPubSubBase / BytesSharedMemory interface (string topics, bytes payloads). Lazily creates pub/sub DataChannel pairs on first publish/subscribe per topic. Also adds: - WebRTCTransport in dimos/core/transport.py (mirrors LCMTransport pattern, no encoding - bytes only). - WebRTC benchmark testcase in dimos/protocol/pubsub/benchmark/testdata.py, gated on aiortc + CF_TELEOP_APP_ID / CF_TELEOP_APP_SECRET env vars. - Integration test in dimos/protocol/pubsub/impl/test_webrtcpubsub.py covering basic pub/sub, latency, and throughput (all live tests skip without CF credentials). - aiortc + httpx as new 'webrtc' optional extra in pyproject.toml. Live benchmark (us-east-2 -> CF edge): - 64-256B: ~10K msgs/s, 0% loss - 1KiB: ~7K msgs/s, 0% loss - >= 64KiB: dropped (above SCTP message size) - Median single-RTT: ~2.5 ms
…sport - Add BrokerProvider: DataChannelProvider that works through the hosted teleop broker (dimensional-teleop) instead of directly with CF credentials. Handles session registration, heartbeat loop, and DataChannel creation when an operator joins via the broker's bridge-datachannel API. - Extend WebRTCTransport with optional msg_type parameter for typed LCM encode/decode with fingerprint-based filtering. Multiple transports can share a single multiplexed DataChannel and each receives only its type. - Add hosted teleop blueprints (dimos/teleop/hosted/) demonstrating the module-free architecture: make_teleop_hosted_go2() uses pure transport (zero modules), make_teleop_hosted_go2_scaled() adds a thin TeleopScalerModule for speed scaling only. - Add unit tests for typed mode, fingerprint filtering, multiplexed dispatch, and BrokerProvider credential validation.
- Rebase on main and regenerate uv.lock (resolve conflict) - Add _LoopbackProvider (in-process, no network) to benchmark testdata - Enables local WebRTC transport benchmarking without CF credentials - All 12 message sizes pass locally (2.78s total)
The previous lock regen dropped the `exclude-newer-span` marker, leaving only the frozen `exclude-newer` timestamp. uv then treats every resolve as "cooldown was newly added" and forces a re-resolve against today minus 7 days — which currently excludes md-babel-py 1.2.0 (published 2026-05-15) and breaks `uv sync --extra all` / `uv lock`. Re-adding the span line tells uv the lock was generated with P7D semantics, so the existing pinned versions are honored.
- Remove __init__.py files (project policy: no init files) - Remove section markers from test_webrtcpubsub.py - Regenerate all_blueprints.py (adds TeleopScalerModule) - Fix WebRTCTransport.__reduce__ to preserve msg_type across pickle - Fix CloudflareProvider.publish() race: snapshot loop ref before use - Fix CloudflareProvider.subscribe() race: check sub_channels inside lock - Add comment clarifying TwistStamped→Twist type safety in blueprint
- Add return type annotations to CloudflareProvider event handlers - Fix type: ignore codes to match actual mypy errors (attr-defined) - Add type annotation to __setstate__ dict parameter - Add type: ignore[arg-type] for WebRTCPubSub→PubSub duck typing in benchmark - Remove TwistStamped subclass comment from blueprint
If createDataChannel raised inside the heartbeat, the id was already recorded, so every later heartbeat skipped the open and the channel stayed dead until the operator id changed. Extract _reconcile_channels and store the id only after a successful open so the next beat retries. Also drop needless forward refs in Blueprint.transports (review nit).
…pping Per Paul's review: a dev or CI env without the webrtc deps used to skip the entire suite invisibly. The guard asserts WEBRTC_AVAILABLE when CI is set; downstream/no-network installs (Sam's case) keep skipping.
Per review: the repo convention moved dev-only/live tests to tool_*.py filenames (never collected; run by explicit path) and removed the tool marker codebase-wide. Cred-gating skipifs stay for keyless path runs.
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #2798 +/- ##
==========================================
+ Coverage 71.91% 71.97% +0.05%
==========================================
Files 979 987 +8
Lines 87346 89117 +1771
Branches 7982 8133 +151
==========================================
+ Hits 62819 64140 +1321
- Misses 22387 22794 +407
- Partials 2140 2183 +43
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
… speaker+livekit tests
ruthwikdasyam
force-pushed
the
ruthwik/hostedteleop/2
branch
from
f3e5502 to
57d37ef
Compare
ruthwikdasyam
changed the title
…nds, review fixes
# Conflicts: # pyproject.toml # uv.lock
…size cap, rewrite hosted docs
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…dio_in on livekit)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Teleoperating a robot over the internet (not just LAN) needs a dial-out architecture: robots behind NAT/cellular can't accept inbound connections, so they must connect out to a broker, and browser/VR operators reach them through it. dimos had no such path — only the LAN-only quest WebSocket teleop.
Closes DIM-XXX
Solution
Robot-side hosted teleoperation over WebRTC, via the dimensional-teleop broker (Cloudflare Realtime or LiveKit SFU). One session per robot carries video, commands, telemetry and safety.
Transport layer - CloudflareTransport / LiveKitTransport (+ *VideoTransport) bind directly to blueprint streams; a per-process BrokerProvider (CF) / LiveKitBrokerProvider owns the single session. Commands, state and video all ride it - no separate teleop module owning its own PeerConnection.
Go2HostedConnection - colocates every broker-bound stream on one module (the driver is dedicated_worker, so they must share a process to share a session). Adds:
Clock-sync ping/pong is answered inline in the provider (no module hop, so RTT/offset stay jitter-free). Legacy HostedTeleopModule is deprecated. Arm/manipulation hosted teleop was split out to a follow-up branch to keep this Go2-focused.
Coverage: unit tests for the command plane, camera mux, clock-sync ping, stream/video stats, speaker track, and the LiveKit frame packing. The live WebRTC session paths are exercised by the opt-in broker e2e/benchmark cases (they need a real SFU + operator), not unit tests.
How to Test
Contributor License Agreement