feat(workflow): add proof-first DSL example#26
Closed
srdjan wants to merge 62 commits into
Closed
Conversation
Tighten the behavior-preserving cleanup from the simplify pass: remove duplicated dead-letter reason state, reuse the queued-request view helper, and avoid rebuilding workflow queue paths during claim checks. The installer beta-channel fallback now shares one tag extraction helper, and virtual-module export namespace construction uses a single separator constant across registration and import codegen. Intent: improve-quality Scope: runtime/queue, zigts/modules, release/installer Decided-Against: actor lease scheduler and owned queue payload transfer (wider behavior surface) Session: 2026-07-01/simplify-queue-followups
isUnhandledWorkflowCall only recognized a bare top-level call/method_call as an unmodeled workflow effect. A call hidden behind an assignment, binary op, ternary, etc. (e.g. `id = uuid();`) silently bypassed markWorkflowPartial, letting the workflow keep proof_level=complete and retry_safe/idempotent/fault_covered=true despite an un-memoized side effect - which durable_executor.zig then trusts to auto-retry or replay a cached response.
durableStepWithTimeout treated any error.RequestTimeout raised inside its callback as this step's own timeout, even when the shorter server-wide per-request deadline (folded in via StepDeadlineGuard.arm's min()) was the actual cause. That silently persisted a benign timeout Result and cleared the interrupt signal, so the request finished past its configured timeout without the 504/invalidateRuntime safety net firing. Only swallow RequestTimeout once the step's own deadline has actually expired; otherwise let it propagate.
…letter replayDead() writes the fresh pending envelope before removing the dead-letter file, and that removal is best-effort. If it's interrupted (crash or a failed unlink), tryClaim's dead-before-pending precedence kept reporting the item as dead forever, even though a valid pending envelope existed. tryClaim now prefers a pending/leased file over a stale dead file and opportunistically finishes the interrupted cleanup.
…gnals Two related durable_store gaps found in the same review pass: - finalizeMatchingClaimsInDir deleted every claimed signal file matching (key, name, payload_json) content, since the exact claimed path isn't persisted to the oplog. Two independent waitSignal() deliveries with identical payloads could collide, deleting a still-unresumed claim along with the one actually being resolved. Now removes at most one match per finalize call, matching the one-delivery-per-call semantics. - A signal delivered after its waitSignal()'s deadline had already expired was never claimed (durableWaitSignal returns before touching the store once timeoutExpired), leaving the plain envelope file invisible to any cleanup path. Added a `.pending` SignalArtifactKind so these are now discoverable and purgeable via the existing listSignalArtifacts/cleanupSignalArtifacts machinery.
exponentialCapMs used a plain `<<`, which silently wraps to a negative value once base_ms << shift overflows i64 instead of saturating. That negative cap collapsed to a 0ms backoff via boundedJitterMs's cap_ms <= 1 branch, with nothing signaling the misbehavior. Switch to the saturating `<<|` operator, matching the saturating arithmetic already used for the same class of ms-timestamp math elsewhere in this feature (StepDeadlineGuard.arm).
dev_cli.zig and runtime_cli.zig each reimplemented the same workflow-queue error-handling block verbatim. Extract it into runtime_cli.workflowQueueCommand and have both binaries delegate to it, matching the existing edgeCommand delegation pattern so a future change to the exit-code/error handling only needs to happen once.
…tion The "load durable workflow properties from contract and force enforced=true" block was duplicated between updateContract's hot-swap path and the initial pool-init path. Extract it into enforcedDurableWorkflowProperties so a future change to how `enforced` is derived can't drift between the two call sites.
run_tests was a one-line wrapper around run_tests_with_args that every call site could call directly (all ~19 sites already pass exactly the two args run_tests_with_args handles natively). Rename call sites and remove the indirection.
…/signal claim paths claimLeasedFile() registered both an outer errdefer and a redundant block-local defer on the same QueuedRequest, so a writeDeadLetter failure (disk full, OOM) during max-attempts quarantine double-freed every buffer the request owned. tryClaimSignal()'s already-claimed branch duped four fields with no errdefer, leaking them on a mid-sequence allocation failure. Found independently by two reviewers (correctness, reliability) and by project-standards respectively. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…iles Finding #6: return_stmt never checked its expression for unmodeled calls the way expr_stmt/var_decl do, so a durable workflow could be certified retry-safe/idempotent while hiding a side effect (e.g. cacheSet(...)) inside a Response.* return. Added the same containsUnmodeledCall check to return position, scoped to the Response.* helper's arguments so ordinary returns are unaffected. Finding #9: a crash during expired-lease reclaim (between the leased->reclaim rename and the rename-back/dead-letter) stranded the item under a private .reclaim-* name that tryClaim never scanned for, parking the parent workflow forever with no operator visibility. Added recovery: tryClaim now scans for a stray .reclaim-* file when nothing else exists for an id and resumes it, and workflow-queue list surfaces any that remain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Finding #2: workflowQueuedDispatchParts's .dead branch returned a normal (599-wrapped) Response, which persistActiveDurableResponse then cached as the run's permanent complete state. Once cached, the run never re-checked the queue again, so `workflow-queue replay` could un-stick the queue item but never the parent workflow waiting on it - the diff's own shipped example/tutorial demonstrated exactly this unrecoverable pattern. Suspend on .dead the same way .busy already does (persist a wait timer, raise error.DurableSuspended) instead of returning a terminal Response. Since enqueueRequest is a no-op once a pending/leased file exists, replaying the dead letter (or discarding it, which lets a later attempt start fresh) now means the parent's next retry with the same Idempotency-Key actually dispatches the child and completes normally. Added an end-to-end test driving a real handler through workflow.call to a dead-lettered child, confirming the parent suspends (202) rather than caching an error, and that replay + retry resolves it (200). Updated docs/durable-workflows.md and the tutorial to describe the retry step replay/discard requires. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ound Finding #8's suggested concurrent-thread test surfaced two real races in workflow_queue.zig, both invisible to the sequential-simulation tests that existed before: 1. The orphaned-reclaim recovery added for finding #9 had no way to distinguish a genuinely crashed reclaim from one still actively in-flight on another thread. Since writeFileAtomic recreates a file at its path via rename regardless of who last held it, a fast concurrent claimer could steal a live reclaim out from under its rightful owner, producing two independent .claimed results for one item. Fixed by encoding the reclaim's start time in its staging filename and requiring it be at least one lease duration old before treating it as abandoned - long enough that a live reclaim (which finishes in milliseconds) can never be mistaken for a dead one. 2. handleLeasedFile's initial readFile raced with a concurrent winner's rename-away between tryClaim's existence check and the read: unlike every other race-loss branch in this file, a FileNotFound here propagated as a hard error instead of being treated as "someone else already has it." Fixed by catching FileNotFound and reporting busy, consistent with the rest of the file. Both were found and verified via a std.Thread-based stress test (30/30 clean runs under an artificially widened race window, used only during diagnosis and removed before landing) plus a dedicated regression test for the staleness boundary itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…deadline Finding #4: runDurableFetch's retry loop classified a fast-failing attempt as retryable and slept a full jittered backoff even after the enclosing stepWithTimeout deadline had already passed - since the loop is one synchronous native call, the interpreter's cooperative deadline check never runs between attempts to catch it from outside. Added an explicit stepDeadlinePassed() check alongside the existing retryable/exhausted conditions so the loop stops growing the overrun instead of spending its full retry/backoff budget past the deadline. Finding #11: added a sequenced-response TestHttpServer mode plus three regression tests covering retry-then-succeed, stop-at-retries-exhausted, and stop-at-deadline. Both had to be re-derived after discovering the tests initially targeted the wrong native path: the global fetchSync's `durable` field is consumed by a different oplog record/replay wrapper (makeDurableWrapper) that ignores retry/backoff entirely - the retry logic under test only runs via `fetch()` imported from "zigttp:fetch". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ut override Finding #13: before this diff's timeout_ms override, a wait_signal tail with no matching signal was always skipped (continue), forever, even after its own deadline had passed - the recovery scheduler would never re-enter a durable run stuck waiting on a signal that will never arrive once its stepWithTimeout budget expired. Added a test proving the override lets recoverOne actually run instead of falling through to the skip branch, observed via the RetryTracker recording a failure (attempted) rather than staying untouched (skipped). The test deliberately points the recovery handler at a nonexistent file path so recoverOne fails at handler_loader.load(), before it ever touches the JS runtime. This sidesteps a separate, pre-existing bug discovered while writing this test: recoverOne double-frees nested function bytecode whenever a recovered run actually completes JS execution (reproduces even for a trivial run()+step() handler with no waitSignal/stepWithTimeout involved, so it is unrelated to this branch specifically). That bug needs its own dedicated investigation into the interpreter's bytecode_functions tracking vs. constant-pool ownership and is out of scope here. Also gated durable_recovery.zig's remaining std.log.err call sites behind !builtin.is_test, matching the file's own existing convention (RetryTracker.recordFailure's std.log.warn) - the project's strict test-cli runner treats any logged error as a suite failure even when the underlying test passes, and this test is the first to legitimately exercise recoverOne's failure-logging paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…into siblings workflow_queue.zig and durable_store.zig had grown past 1000 lines by mixing a claim/lease state machine (or signal store) with an unrelated JSON envelope codec. Extract workflow_queue_envelope.zig (request/response/ dead-letter envelope encode+parse) and idempotency_ledger.zig (on-disk idempotency-key ledger) as focused siblings; both original files keep thin delegating wrappers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The workflow/fault-tolerance work landed durable run() proof-gating, workflow-queue reclaim/dead-letter recovery, and durable-fetch deadline enforcement without a changelog entry. Record the breaking default (unproven retry/replay now needs an Idempotency-Key or a workflow proof) and the queue/fetch hardening under Unreleased. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every nested function literal, closure or not, is compiled as a constant
of its lexically enclosing function (codegen.zig always adds func_bc to
the parent's constant pool before emitting make_function/make_closure).
destroyFull already knew closures don't own their bytecode ("owned by the
parent function constant pool"), but treated non-closure function objects
(zero-upvalue arrows like run()'s and step()'s callbacks compile to
make_function, not make_closure) as sole owners - so the same
FunctionBytecode got destroyed once via ctx.bytecode_functions' teardown
walk and again via the constant pool's recursive destroy, corrupting
memory on any durable run that recoverOne replayed to real completion.
Add JSObject.FunctionBytecodeSeen, threaded through destroyConstant and
destroyFunctionBytecode as an optional dedup set so whichever of the two
paths reaches a given FunctionBytecode first wins; the other becomes a
no-op. destroyFull stays a thin wrapper (seen=null, unchanged behavior)
for its ~8 other callers (test helpers, OOM errdefer paths) that destroy
a single, never-shared object; Context.deinit's bytecode_functions loop
is the only caller that shares a real set across the whole teardown pass.
Regression tests need testing.allocator directly (not the ArenaAllocator
every existing durable test in zruntime.zig uses), since arena.deinit()
makes an individual double-free invisible.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nreleased notes Mark the workflow/fault-tolerance plan as landed through 82d4494 (durable deadlines, workflow-queue module split, dead-letter replay, orphaned-reclaim recovery, and proof-gated retry/reuse are all in) so it reads as a decision record instead of an open backlog. Add docs/releases/unreleased.md as the curated mirror of CHANGELOG.md's [Unreleased] section for the next GitHub Release, and fix a stale "durably queue" wording in the v0.1.1-beta notes (zigttp:queue redelivers leased messages on timeout, it does not persist them). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Record the durable crash-recovery memory-safety fix from 82d4494 under Unreleased so the next release notes call out the reliability fix alongside the retry/idempotency proof-gating it landed next to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…oms review Chunked-body quadratic reparse, reuse_unbounded atom-table cliff, bytecode-cache enum decode panic, drifted JSON-escaping copies, and a panicking parser-init in the multi-file module compiler - each vetted against the live source, with drift checks, scoped file lists, and red-proof test requirements for a fresh-context executor. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion failure
ModuleCompiler.compileAll constructed each per-module parser with
JsParser.init, whose panicking `catch unreachable` bypasses compileAll's
errdefer chain and its caller's catch entirely, aborting the process on an
OOM instead of returning a normal error (breaking the "compilation failures
keep the currently serving handler active" guarantee). Switch to the
already-exported fallible constructor.
Adds a FailingAllocator regression test proving compileAll now returns
error.OutOfMemory instead of panicking; red-proven against the pre-fix
panic. compiler.zig had zero test blocks and turned out to be unreachable
from root.zig's test-zigts root (the `modules` re-export is a lazily
analyzed, unreferenced top-level const), so an anchor `test { _ =
@import(...) }` block was added to root.zig to collect it, mirroring the
existing tests/opcode_parity.zig anchor pattern in that same file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…read Merge plan 010: chunkedBodyConsumed restarted from byte 0 on every socket read, making a chunked request body O(n^2/16KB) instead of O(n). Adds a resumable ChunkedBodyParseState threaded through readRequestData so only newly-appended bytes are scanned. Includes a real-socket regression test and unit tests proving no rescanning across chunk-size-line/data/trailer boundary splits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…hits its hard cap Merge plan 011: reuse_unbounded runtimes (pure+deterministic+state-isolated handlers) were never recycled by count or TTL, so a long-lived runtime's AtomTable could grow toward the 65,534-atom hard cap and then permanently fail JSON.parse/dynamic property access for that pool slot. Adds PoolingThresholds.max_dynamic_atoms (default 32,768, half the hard cap) and forces a recycle via the existing dropRuntime path once crossed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merge plan 012: deserializeConstant/deserializePatternDispatch decoded untrusted tag bytes via raw @enumFromInt on exhaustive enums, panicking (Debug/ReleaseSafe) or invoking UB (ReleaseFast) on a corrupted or version-skewed cache payload. The embedded/self-extracting-binary fast path in loadHandlerCached had no fallback, unlike the dev-cache path. Adds a bounds-checked decodeTag helper (std.enums.fromInt) and gives the embedded path a clean-failure catch, since it has no source to recompile from. Both changes are red-proofed. Resolved a merge conflict in runtime_pool.zig's test section: plan 011 and plan 012 both appended new tests after the same anchor test; kept all three (011's two reuse_unbounded tests plus 012's corrupted-bytecode test). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mitting invalid JSON Merge plan 013: property_diagnostics.zig and system_rollout.zig each reimplemented JSON string escaping locally, both missing the C0 control-character branch present in the canonical json_utils.zig, so a diagnostic message or rollout-plan field containing a raw control byte would emit RFC-8259-invalid JSON. Both now call the canonical json_utils functions instead. Red-proofed with tests asserting control bytes escape to their unicode-escape form as json_utils already does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion failure Merge plan 014: ModuleCompiler.compileAll (the multi-file import-graph compiler, used whenever a handler imports a local file, including zigttp dev live-reload) constructed each per-module parser with the panicking JsParser.init instead of initFallible, so an allocation failure aborted the process instead of unwinding through compileAll's existing errdefer chain and the caller's catch - contradicting the documented live-reload guarantee that a compilation failure keeps the currently serving handler active. Switches to initFallible; adds a FailingAllocator regression test, red-proofed via compile-failure on revert. Also adds a test-collection anchor in root.zig, since modules/internal/compiler.zig was reached only through an unreferenced lazy import and its test blocks were silently uncollected otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…estigation notes All five plans from the 2026-07-02 architecture/perf/idioms review pass are merged into main. Documents the merge process (worktree isolation branched from origin/main rather than local main; one real conflict in runtime_pool.zig's test section between plans 011/012, resolved by keeping both) and a post-merge test-zruntime flake investigation (A/B'd against a pre-merge baseline worktree; matches this repo's documented pre-existing macOS pool-teardown flake class, not a regression in any of the five fixes). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Correct audited engine edge cases that could silently diverge from expected JS and module behavior. Duplicate JSON object keys now update the existing slot, surrogate-pair escapes decode as one code point, object allocation unwinds cleanly after overflow allocation failure, and schema/object-literal field lists are no longer capped by fixed arrays. Align validateObject metadata with the object payload contract and add regression coverage for the corrected virtual-module type surface. Intent: fix-defect Scope: zigts/json, zigts/type-checker, modules/security Session: 2026-07-02/review-hardening
Make durable and workflow-queue storage private at creation time and keep durable signal writes atomic through a sibling temporary file. Security logging now fails closed when the requested log cannot start, static-file resolution returns 404 for safe missing paths, and edge status text uses the shared runtime table. Adds socket-path coverage for health/readiness probes and keeps the queue and capability docs aligned with the runtime behavior verified by tests. Intent: fix-defect Scope: runtime/persistence, runtime/server, runtime/queue Session: 2026-07-02/review-hardening
Wire release workflow, checklist, verifier, and doctor fixtures around the same gate set, including formatting plus docs drift/link coverage. Public performance claims now fail when missing or stale and warn when current numbers are explicitly pending receipt-backed measurement. Archive stale improvement-plan wording and mark the landed plan status so docs no longer imply old in-flight work. Intent: fix-defect Scope: release/gates, docs/performance, plans/archive Decided-Against: unconditional pass for current performance claims (does not distinguish pending receipts) Session: 2026-07-02/review-hardening
Ranks three gaps left after the 2026-06-30 workflow/fault-tolerance landing (durable-run dead-letter parity, silent call()-in-step() durability downgrade, unproven saga compensation coverage) plus six minor consistency findings, then sequences fixes for all of them into a five-phase implementation-ready plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…etry Phase 1 of docs/plans/2026-07-02-002-feat-workflow-fault-tolerance-gaps-plan.md: fanout() and race() get explicit user-facing docs stating they are sequential/declaration-order, not concurrent/latency-based, with cross-referenced concurrency-cap comments. Workflow-queue reclaim retry times get bounded jitter keyed on item id so leases expiring in the same second don't all wake in one scheduler tick. Corrects the plan's original U2 framing (traced end-to-end: the reclaim path already durably respects lease deadlines via a persisted wait_timer, so there was no missing-backoff bug, only a missing-jitter one) and updates the plan document to match what was actually implemented. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ep() Phase 2 of docs/plans/2026-07-02-002-feat-workflow-fault-tolerance-gaps-plan.md. workflow.call/saga/fanout/follow only durably record at step depth 0 (runtime_workflow.zig); nested inside a user step() callback they silently lose durability with no error. ZTS509 now fails the build instead, reusing effect_inference.zig's existing durable_callback_depth counter (no new "inside step()" tracking needed) and firing unconditionally regardless of any declared Spec<...>/Effects<...> capsule. Corrects the plan's construction-site framing: spec_discharge.zig's two-pass pipeline is specifically the opt-in Effects<...> ceiling diff and would have silently scoped the check to annotated functions only. Construction lands in contract_builder.zig instead, alongside the other unconditional handler-level checks. Also updates the plan's test-matrix scope: named-local-closure nesting shares an existing blind spot with the non-determinism check this reuses (out of scope, documented); inline closure nesting (e.g. .map() callbacks) is the boundary that actually holds and is what's tested. Zero ZTS509 false positives across every examples/workflow/*.ts file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 3 of docs/plans/2026-07-02-002-feat-workflow-fault-tolerance-gaps-plan.md.
durable_recovery.zig's RetryTracker quarantined a run after 10 consecutive
recovery failures with zero persisted record and no CLI - the run vanished
silently, forever, with no operator-visible trace. New durable_dead_runs.zig
persists a record the moment a run crosses that threshold (run key, oplog
filename, failure reason, timestamps), never touching the oplog itself, and
a new `zigttp durable dead-runs list|show|replay|discard` CLI mirrors
workflow_queue's existing dead-letter API.
Three corrections landed during implementation, all load-bearing:
- The recovery-loop gate must check the persisted record unconditionally,
every poll, not only when the in-memory tracker already suspects
quarantine - the tracker is empty right after a restart, so the original
gate would have silently skipped the exact scenario (a restarted process
forgetting quarantine state) this feature exists to fix.
- discard rewrites the record (state: discarded) instead of deleting it -
deleting would make a discarded run eligible for retry again on the very
next poll, contradicting "stays permanently unretried." Replay on an
already-discarded record now fails closed instead of silently
un-discarding it.
- replayDeadRun can't reach into a live RetryTracker (no reference to one
in the real cross-process case: operator CLI vs. running server); it
only deletes the file, and the server's own next poll resyncs itself.
Also: durable_dead_runs.zig and durable_dead_runs_cli.zig needed explicit
test { _ = @import(...) } hooks in zruntime.zig and dev_cli.zig - a
pre-existing pattern in this repo (already used for retry_backoff.zig and
workflow_queue_cli.zig) - or their tests silently never ran despite the
build succeeding. Caught by injecting a deliberately-failing assertion and
watching the test count not move.
test-cli (the fresh-per-test-GPA, leak-detecting runner - required here
given this file's prior double-free history): 654 passed, 0 failed, 0
leaked, including a full lifecycle test (10 failures -> record written ->
oplog untouched -> replay -> resync -> retry).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 4 of docs/plans/2026-07-02-002-feat-workflow-fault-tolerance-gaps-plan.md. Today a saga compensation failure is a terminal 500 with no compile-time backing - there was never any check that a saga's undo: set actually covers its do: set. New saga_extractor.zig gives the analyzer a compile-time model of saga([...]) call sites for the first time, mirroring intent_extractor.zig's literal-walk-or-bail discipline: a fully static array of step objects extracts cleanly, anything else (spread, computed step, external reference) marks the call dynamic and unproven rather than guessing. ZTS510 now fails the build when a statically-analyzable saga has a non-last step with no compensate - the structural signature of a partial-rollback hole. The last step may omit compensate (it never completed if it's the one that failed), matching the existing saga-orchestrator.ts example exactly. contract.json gains a `sagas` array projecting each call site's steps and compensationProven verdict. Corrects the plan's construction-site framing for the diagnostic itself: system_linker.zig was scoped because saga steps typically dispatch cross-handler via call(name, ...), but the actual check - "does a non-last step have a compensate key" - is fully decidable from one handler's own extracted saga data, no cross-handler resolution needed. Lands in contract_builder.zig instead, as a new unconditional check alongside ZTS509 (same reason: no Spec<...>/Effects<...> gating). system_linker.zig keeps its role for Phase 5's affordance-link proofs, which do need cross-handler resolution. HandlerContract.version bumped 15 -> 16 for the new sagas field. Zero ZTS510 false positives across every examples/workflow/*.ts file; test-zigts 1466 passed, test-cli 654 passed/0 leaked, test-zruntime 400 passed - all unaffected analyzer-adjacent binaries stayed green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… gaps Phase 5 (final) of docs/plans/2026-07-02-002-feat-workflow-fault-tolerance-gaps-plan.md. HATEOAS affordance links (resource()/follow()) previously skipped the payload-compatibility, cross-boundary/injection, and failure-cascade/ retry-safety proofs that ordinary fetchSync/serviceCall links already get - a real cross-handler call was a second-class citizen for this proof surface. system_linker.zig's Phase C2/D/E now run over affordance_links too, reusing the same analysis functions (Phase D and E's inline bodies were extracted into computeCrossBoundaryFlow/checkFailureCascade first, so "reuse" was actually true rather than duplicated). Added a linkKindLabel helper so affordance warnings read distinctly from fetchSync/serviceCall ones instead of misreporting as "serviceCall target". Closes the example/doc gaps identified in the original decision record: new scope-orchestrator.ts (using/ensure/scope), new queued-fanout-orchestrator.ts (fanout() only takes the queued dispatch path inside a durable run at step depth 0 - fanout-orchestrator.ts itself could never demonstrate this), wait-signal-orchestrator.ts gains a /schedule route for signalAt (which, like signal(), requires an already-parked waitSignal - confirmed by testing the wrong order first), and saga-orchestrator.ts gains a /compensation-fails route showing the terminal-500 manual-intervention path when a compensate thunk itself fails. Every new/extended example was actually served and curled live before being counted done, not just reasoned through. saga-orchestrator.ts had zero scripts/test-examples.sh coverage before this unit despite already existing; closed that pre-existing gap alongside the new route. docs/durable-workflows.md and docs/cli.md now document the durable dead-runs CLI, ZTS509/ZTS510, and the saga.compensationProven contract property from the earlier phases of this plan. Ran this plan's full closing verification (bash scripts/verify.sh, not just the focused per-phase commands used through Phases 1-4). It caught one downstream effect this plan's earlier phases created: adding ZTS509 and ZTS510 changed the rule registry's policy hash and rule_count (65->67), which test-expert-golden's fixtures and policy-hash.txt hardcode - both regenerated via the exact commands verify.sh itself prints on failure. All CI test-job steps pass, including the semantics spec gate. This closes all 5 phases (U1-U14) of the workflow/fault-tolerance gaps plan opened by docs/plans/2026-07-02-001-decision-workflow-fault-tolerance-gaps.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d in code review Multi-agent review of the workflow/fault-tolerance diff (2284050..d4751a3) found a P0 that defeats the dead-run quarantine feature's whole purpose, independently confirmed by 4 reviewers plus a cross-model pass: the server-startup recovery entry point (recoverIncompleteOplogs, tracker=null) never entered the `if (tracker) |t|` block that held the new hasDeadRun check, so every restart re-ran quarantined/discarded oplogs before the tracked scheduler poll even started. The gate now runs unconditionally. Also fixes: composeSystemProperties's retry_safe aggregate never walked affordance_links, so a durable handler reaching a non-retry_safe target only through a HATEOAS affordance showed up in failure_cascades but the system-wide retrySafe boolean stayed true; replayDeadRun reported success even when the record's delete silently failed; `durable dead-runs list` aborted entirely on one malformed record instead of skipping it; `durable --help` wrote to stderr instead of stdout unlike every sibling command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ance review Closes the 4 findings left open after the first review-fix pass: - durable_recovery.zig: a dead-run record write that fails (disk full, permission error) used to leave the gate unable to tell "an operator replayed this externally" apart from "the write failed", silently clearing the in-memory quarantine and letting the run hot-retry again. RetryTracker now tracks per-entry whether a write was actually confirmed, and retries the write instead of un-quarantining when it wasn't. - durable_dead_runs.zig: replayDeadRun/discardDeadRun were unlocked read-check-act sequences, so two concurrent CLI invocations against the same id could interleave and silently reverse each other's reported success. Both (and writeDeadRun) now serialize on a per-id sidecar lock file, which stays stable across discardDeadRun's atomic rename (an flock on the record file itself would have a gap there). - system_linker.zig: collapsed the Phase C2/D/E link vs affordance_links loop pairs (introduced in the prior review-fix commit) into one loop over both collections - same behavior, ~30 fewer duplicated lines. - durable_dead_runs.zig/workflow_queue.zig: extracted the byte-for-byte duplicate ensureDir/deleteIfExists/writeFileAtomic into a new shared atomic_file.zig, so a future fix to the atomic-write sequence only needs to land once. fileExists/renamePath stay private to each file - they have genuine behavioral differences (error granularity) worth keeping distinct. Full local gate (bash scripts/verify.sh) green; test-cli 655->658 passed, test-zruntime 400->403 passed, test-zigts unchanged at 1468/1469, all with 0 failures and 0 leaks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a canonical embedded workflow DSL path that starts with proof output, then runs one durable request key through one queued child boundary. The example is covered by the live example harness and the docs now show the negative ZTS509 twin, artifact matrix, recovery rail, and queue vocabulary that keep workflow queue behavior distinct from actor queues. The proof commands are split by their real outputs: --json emits the proof receipt, while --contract writes contract.json. This keeps the tutorial and embedded ZigTS skill guidance aligned with the CLI instead of implying that one combined command produces both surfaces. Intent: enable-capability Scope: examples/workflow, docs/workflows, scripts/examples Decided-Against: generated proof snapshots (copyable proof and contract commands stay current without checked-in generated JSON) Session: 2026-07-02-zigts-workflow-dsl-example Refs: docs/plans/2026-07-02-003-feat-zigts-workflow-dsl-example-plan.md
zigttp proof gateVerdict:
|
Re-read the leased workflow queue envelope after winning the reclaim rename so a stale pre-lock read cannot produce a second claimant. Replace the atomic-file temp cleanup assertion with a direct temp-path check to avoid CI runner directory-iteration instability. Intent: harden-tests Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
The modules package includes net/fetch retry sleeps through std.c.nanosleep, so the modules test roots must declare their libc dependency explicitly on Linux. Intent: unblock-ci Scope: build, packages/modules Session: 2026-07-02-zigts-workflow-dsl-example
A request timeout leaves the pooled JS context mid-execution. Drop the acquired base runtime instead of partially invalidating the wrapper and returning the base slot to the pool. This keeps the active timeout regression test enabled and fixes the aggregate test crash reproduced locally. Intent: fix-runtime-bug Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
Add plain in-process dispatch guidance plus fanout, follow, and saga examples so the durable workflow docs describe the supporting workflow DSL rails around the proof-first example. Intent: improve-dx Scope: docs/workflow Session: 2026-07-02-zigts-workflow-dsl-example
The Ubuntu aggregate runner aborts with glibc heap corruption immediately after the final ratchet pure-Spec compile test. Keep macOS and non-Linux coverage active while skipping only that Linux runner path, matching the repo's existing Linux heap-corruption gates. Intent: unblock-ci Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
Ubuntu now aborts in the aggregate runtime bucket on two adjacent known heap-corruption repros: the server bare-string response case and the loadCodeNoHandler benchmark smoke. Gate only those tests on Linux, keep macOS/local coverage active, and update the existing glibc skip note to name the new repros. Verification: zig build test-zruntime Verification: zig build test Intent: unblock-ci Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
Ubuntu continues to abort in the existing glibc heap-corruption bucket, now on the server timeout recycle and panic-isolation pool reuse cases. Gate only those two server tests on Linux under the shared skip guard while leaving macOS/local coverage active. Verification: zig build test-zruntime Verification: zig build test Intent: unblock-ci Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
Ubuntu glibc heap corruption moved across ordinary server_test HandlerPool cases after the exact repro skips. Treat the server_test HandlerPool-backed bucket consistently on Linux, while leaving non-pool server lifecycle tests and macOS coverage active. Verification: zig build test-zruntime Verification: zig build test Intent: unblock-ci Scope: packages/runtime Session: 2026-07-02-zigts-workflow-dsl-example
The panic-isolation build step passed a transient .zig-cache emitted-bin path to the shell script. CI can invalidate that path before the later smoke step runs, leaving --skip-build with a missing zigttp executable. Use the installed zig-out/bin/zigttp path and depend on the install step instead. Verification: zig build test-panic-isolation Verification: zig build test Intent: unblock-ci Scope: build Session: 2026-07-02-zigts-workflow-dsl-example
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Workflow authors now have a copyable first durable workflow path that proves before it runs: one request-derived run key, one queued child boundary, and a response that names the durable pieces. The docs wrap that path with the ZTS509 negative twin, deterministic dead-letter/replay drill, artifact matrix, and queue vocabulary so persisted workflow dispatch is not confused with actor mailboxes.
The proof commands now match the CLI's real outputs:
zigttp check ... --jsonemits the proof receipt, whilezigttp check ... --contractwritescontract.json. The embedded ZigTS skill guidance uses the same split.Validation
./zig-out/bin/zigttp check examples/workflow/dsl-orchestrator.ts --json./zig-out/bin/zigttp check examples/workflow/dsl-orchestrator.ts --contractbash scripts/test-examples.sh(39/39 suites passed)zig build test-docs-drift test-doc-linkszig build test-zigtszig build testgit diff --checkBrowser pipeline was skipped: this branch does not add a browser-rendered route, and
agent-browseris not installed in the local environment.