feat(acp): add dream memory consolidation — dispatch, skill distribution, relay sweep#1365
Draft
wpfleger96 wants to merge 6 commits into
Draft
feat(acp): add dream memory consolidation — dispatch, skill distribution, relay sweep#1365wpfleger96 wants to merge 6 commits into
wpfleger96 wants to merge 6 commits into
Conversation
Implement Phase 2 of the dream skill: harness-side dispatch logic for memory consolidation turns. Dream is lowest-priority (pending work > heartbeat > dream), preemptible by any inbound event, and follows the heartbeat session lifecycle (reuse until invalidated on cancel). Key additions: - KIND_DREAM_DUE (24300) ephemeral event constant in buzz-core - PromptSource::Dream variant with full match coverage - Dream session management (create once, reuse, invalidate on cancel) - dispatch_dream() with control_tx for preemption - Dream-due event detection sets pending flag (dispatch on next tick) - cancel_in_flight_dream() fires Cancel on any accepted inbound event - Panic recovery clears dream_in_flight via is_dream on TaskMeta - load_dream_prompt() reads .agents/skills/dream/SKILL.md at startup Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Implements the full dream consolidation system across four areas: **1. Dispatch liveness (Thufir IMPORTANT #1)** Extracts `maybe_dispatch_dream()` helper, called at four state-transition points: after KIND_DREAM_DUE receipt, after heartbeat fires (idle gap), after prompt result + drain, after panic recovery + drain. Removes the old else-if chain that made dream reachable only when a heartbeat was already in flight — fixing both starvation modes: - Single-agent pool + fast heartbeat: dream now fires in the idle gap after heartbeat completes, not waiting for a concurrent heartbeat slot. - Heartbeat disabled: dream fires immediately on dream-due receipt. Guard: no flushable queue work, dream_pending, !dream_in_flight, !heartbeat_in_flight, idle agent. Preserves pending > heartbeat > dream priority. **2. Skill distribution via Nest scaffolder (Will's requirement)** Adds nest_dream_skill.md (include_str!'d as DREAM_SKILL_MD), written to .agents/skills/dream/SKILL.md at Nest init, mirroring the buzz-cli skill pattern (nest.rs:42-174). Includes NEST_DREAM_SKILL_VERSION=1 for refresh-on-bump, refresh_dream_skill_md_if_stale() with atomic tempfile write, and dream skill symlinks in ensure_skill_symlinks() for all known provider dirs (.goose/skills, .claude/skills, etc.). This makes load_dream_prompt()'s cwd-relative read correct — the Nest IS the cwd, and the scaffolder now guarantees the file. **3. Missing prompt is startup-visible error (Thufir IMPORTANT #2)** dispatch_dream() now logs tracing::error! (target: "dream") when the prompt is None, returns the claimed agent, clears dream_pending so it does not retry on every signal, and documents this as a configuration error (missing SKILL.md). No more silent drop. **4. Relay sweep + dream-due emission (Phase 3)** In buzz-relay: - Config: dream_memory_budget_bytes (default 65536, env BUZZ_DREAM_MEMORY_BUDGET_BYTES), dream_sweep_interval_secs (default 300, env BUZZ_DREAM_SWEEP_INTERVAL_SECS). Set budget to 0 to disable. - main.rs: spawns sweep task if budget > 0 and deployment_community is Some. Each interval: queries agents_over_memory_budget, idle-gates each via buzz_pubsub::presence::get_presence (no live Redis key = idle), builds KIND_DREAM_DUE ephemeral event tagged #p=[agent_pubkey], signs with relay keypair, publishes via pubsub. - Staleness ceiling satisfied: presence TTL is 90s, default sweep is 300s, so "no presence key" already implies > 90s silence. In buzz-db: agents_over_memory_budget() aggregates SUM(LENGTH(content)) for non-tombstone kind:30174 engrams per community, returns agents exceeding budget. **5. Tests (Phase 4)** Six unit tests in dream_dispatch_tests cover: - Guard: dream_pending=false → no dispatch, agent not consumed. - Guard: heartbeat_in_flight=true → blocked (priority order). - Regression: heartbeat disabled → dream still dispatches. - Guard: empty pool → pending preserved for next idle gap. - Fix 3: missing prompt → pending cleared, agent returned, no task. - Guard: dream_in_flight=true → no double dispatch. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
changed the title
Addresses three Thufir pass-1 review findings:
[CRITICAL] ACP now subscribes to KIND_DREAM_DUE
The relay sweep emits KIND_DREAM_DUE as a global #p-tagged ephemeral event.
The old code relied on the membership subscription (hard-coded to only
KIND_MEMBER_ADDED_NOTIFICATION + KIND_MEMBER_REMOVED_NOTIFICATION) to deliver
it — a false assumption. The event was never received.
Fix: dedicate a DREAM_SIGNAL_SUB_ID subscription (kinds: [KIND_DREAM_DUE],
#p=[agent_pubkey], no #h). Infrastructure mirrors the existing membership and
observer-control subscriptions:
- DREAM_SIGNAL_SUB_ID = "dream-signal" const in relay.rs
- SubscribeDream RelayCommand variant
- dream_sub_active flag in BgState
- send_dream_subscribe() + build_dream_req_filter() helper functions
- Routing: forwards BuzzEvent { channel_id: Uuid::nil(), event } for
KIND_DREAM_DUE events — nil sentinel is safe because lib.rs checks
kind_u32 == KIND_DREAM_DUE before any code touches channel_id (continue)
- CLOSED handler resubscribes or triggers reconnect
- resubscribe_after_reconnect() restores dream sub after socket loss
- subscribe_dream_signals() public method on HarnessRelay
- Called in lib.rs right after subscribe_membership_notifications() at startup
No watermark/replay needed: dream-due events are ephemeral relay-initiated
signals. A missed one is safe — the relay sweep re-emits on the next interval.
[IMPORTANT] Honest idle gate comment with design rationale
The relay-side idle gate is intentionally a coarse pre-filter: no live presence
key = no heartbeat in 90s = probably idle. This is by design, not an oversight.
The real over-fire protection is agent-side: maybe_dispatch_dream() requires no
flushable queue work, no heartbeat in flight, and an idle agent in the pool;
dream runs at lowest priority and is preemptible via ControlSignal::Cancel.
A false "idle" from the relay costs nothing — the agent ignores it or runs a
preemptible low-priority turn.
The idle gate logic is extracted into agent_is_idle(presence: Option<&str>) -> bool
in buzz-relay/main.rs — a pure function that makes the binary decision testable
without a Redis integration harness.
No last_turn_ended_at column exists in the DB; a timestamp-based gate is not
implementable. Presence-TTL (90s) is the only available signal.
[IMPORTANT] Subscription filter tests + idle gate boundary tests
Six tests in relay::dream_subscription_tests pin the NIP-01 filter contract:
- KIND_DREAM_DUE is present in kinds
- Exactly one kind (no accidental broadening)
- #p contains the agent pubkey
- No #h (global, not channel-scoped)
- Different pubkeys produce different filters
- DREAM_SIGNAL_SUB_ID is distinct from MEMBERSHIP_NOTIF_SUB_ID and
OBSERVER_CONTROL_SUB_ID
Four tests in relay::tests cover the idle gate:
- live "online" presence blocks
- live "away" presence blocks
- any non-None status blocks
- None (expired/absent) allows
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
KIND_DREAM_DUE (24300) was absent from P_GATED_KINDS, so p_gated_filters_authorized() never enforced #p-must-match-self for it. Any authenticated relay user could subscribe with kinds:[24300], #p:[victim_pubkey] and receive another agent's dream-due signals, leaking "agent X is over memory budget and idle" to anyone who knows an agent pubkey. The kind's own doc already states "single-delivery to the authenticated agent"; P_GATED_KINDS is the registry that makes the relay enforce it. Fix: add KIND_DREAM_DUE to P_GATED_KINDS. Ephemeral kinds are included in this list for filter-layer enforcement only and are never stored, so no schema/migration/tsvector change is needed. The ACP harness's own subscription (DREAM_SIGNAL_SUB_ID, #p=[self]) continues to be accepted — the self case is exactly what P_GATED_KINDS allows through. Test: dream_due_subscription_requires_matching_p_tag verifies: - no #p → rejected - #p:[other] → rejected - #p:[self] → accepted Confirmed: test fails without the registry add and passes with it. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Brings the branch current with main (25 commits) before CI green run. Auto-merged — no conflicts. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Three CI failures on head 30e25b4: Rustfmt: run `just fmt` — collapses no_p let in req.rs:1341, the publish_event() call in main.rs:747, alpha-orders test use imports in main.rs:1020, and reflowed lines in relay.rs + event.rs that fell over the column limit. E0063 OwnedAgent missing field: the branch was 25 commits behind main after #887 (config-bridge) added model_overridden: bool to OwnedAgent. The dream test dummy_agent() helper at lib.rs:4593 didn't set it. Fix: add model_overridden: false, matching the adjacent field pattern. nest.rs file-size override: dream-skill scaffolding (+76 lines) pushed nest.rs to 1526 (gate-count) over the prior 1450 ceiling. Bumped the override in desktop/scripts/check-file-sizes.mjs with a one-line rationale comment noting the dream-skill addition mirrors the buzz-cli seam already in the module; queued to split with the rest of the list. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
force-pushed
the
duncan/dream-harness
branch
from
c38e46d to
5a94a19
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the full dream consolidation system, consolidating Phases 2-4 into a single PR. Fixes two IMPORTANT correctness findings from Thufir's Phase 2 review.
Changes
Fix: dispatch liveness (Thufir IMPORTANT #1)
Extracts
maybe_dispatch_dream()helper called at four state-transition points: afterKIND_DREAM_DUEreceipt, after heartbeat fires (idle gap), after prompt result + drain, after panic recovery + drain.Removes the old
else ifchain that made dream dispatch reachable only when a heartbeat was already in flight — fixing both starvation modes:heartbeat_interval_secs == 0): dream fires immediately ondream-duereceipt.Guard: no flushable queue work,
dream_pending,!dream_in_flight,!heartbeat_in_flight, idle agent. Preservespending > heartbeat > dreampriority.Fix: missing prompt is startup-visible error (Thufir IMPORTANT #2)
dispatch_dream()now logstracing::error!(target:"dream") when the prompt isNone, returns the claimed agent, clearsdream_pendingso it does not retry on every signal, and documents this as a configuration error. No more silent per-signal drop.Skill distribution via Nest scaffolder
Adds
nest_dream_skill.mdalongsidenest_skill.md, wired up identically to the buzz-cli skill:DREAM_SKILL_MDconst (include_str!fromnest_dream_skill.md)..agents/skills/dream/SKILL.mdat Nest init (create-new, idempotent).NEST_DREAM_SKILL_VERSION = 1+refresh_dream_skill_md_if_stale()for refresh-on-bump (atomic tempfile write).ensure_skill_symlinks()for all known provider dirs (.goose/skills,.claude/skills, etc.).This makes
load_dream_prompt()'s cwd-relative read correct — the Nest is the cwd, and the scaffolder guarantees the file.Relay sweep + dream-due emission
In
buzz-relay:Config:dream_memory_budget_bytes(default 65536,BUZZ_DREAM_MEMORY_BUDGET_BYTES),dream_sweep_interval_secs(default 300,BUZZ_DREAM_SWEEP_INTERVAL_SECS). Set to 0 to disable.main.rs: spawns sweep task if budget > 0 anddeployment_communityisSome. Each interval: queriesagents_over_memory_budget, idle-gates viabuzz_pubsub::presence::get_presence(no live Redis key = idle), buildsKIND_DREAM_DUEephemeral event tagged#p=[agent_pubkey], signs with relay keypair, publishes via pubsub.Staleness ceiling satisfied: presence TTL is 90s, default sweep is 300s — no presence key implies > 90s silence, well above 2× sweep gap.
In
buzz-db:agents_over_memory_budget()aggregatesSUM(LENGTH(content))for non-tombstone kind:30174 engrams per community, returns agents exceeding budget.Tests
Six unit tests in
dream_dispatch_testscover:dream_pending=false→ no dispatch, agent not consumed.heartbeat_in_flight=true→ blocked (priority order enforced).heartbeat_interval_secs == 0starvation fix).dream_in_flight=true→ no double dispatch.Files changed
crates/buzz-acp/src/lib.rs— dispatch liveness, missing-prompt error, dream testscrates/buzz-db/src/event.rs+lib.rs—agents_over_memory_budgetquerycrates/buzz-relay/src/config.rs+main.rs— sweep task + config fieldsdesktop/src-tauri/src/managed_agents/nest.rs— skill scaffoldingdesktop/src-tauri/src/managed_agents/nest_dream_skill.md— dream SKILL.md content