Skip to content

feat(testing): per-execution worker serialization#516

Draft
yaythomas wants to merge 1 commit into
pr1/invoker-state-checkpoint-stampingfrom
feat/per-execution-worker
Draft

feat(testing): per-execution worker serialization#516
yaythomas wants to merge 1 commit into
pr1/invoker-state-checkpoint-stampingfrom
feat/per-execution-worker

Conversation

@yaythomas

Copy link
Copy Markdown
Contributor

What

Replace the per-ARN lock with a per-execution worker that runs every operation for one execution one at a time on its own serial lane. Serialization and completion ordering are now structural properties of the design.

Changes

Worker model (worker/ package):

  • SerialTaskLane: single-consumer FIFO queue with one daemon thread. Tasks run in submit order with no overlap.
  • ExecutionWorker: runs every operation for one execution on its lane. Carries invocation state from each task into the next. Tears down on terminal status.
  • ExecutionRegistry: owns one worker per execution ARN. Creates on first use, drops on completion.
  • ExecutionTask / TaskOutcome: contract for work run on a worker. Exclusive access to execution state for the duration of execute.
  • CallableTask: lets existing methods serialize on the lane without restating as a task.

Shared checkpoint write-transaction (checkpoint/core.py):

  • CheckpointCore.apply is the single write-transaction shared by the web and in-process paths (extends match_cached from PR 1).
  • Checkpoint lifecycle actions returned as effects (checkpoint/effects.py) applied after the write. The reentrant notifier is removed.

Invocation runs off the lane:

  • The blocking Lambda invoke runs off the worker lane so handler HTTP callbacks can run on it.
  • Begin (claim gate) and finish (record response, release gate) run as tasks on the lane.

Store coverage:

  • Runner accepts an optional store parameter for store-injection testing.
  • FileSystemExecutionStore writes saves atomically (temp file + replace).
  • New tests/e2e/store_backed_lifecycle_test.py drives a full lifecycle on memory, filesystem, and sqlite.

Documentation:

  • docs/architecture.md: new, describes the worker model, checkpoint flow, invocation state machine, and pagination.

Testing

  • 1333 passed, 95.03% coverage (gate 95%)
  • Typecheck clean (116 files)
  • Format clean
  • Concurrency stress: scale=1 at 16 and 32 way parallelism produces exactly the 6 CHAINED_INVOKE failures and no new failures.
  • Worker concurrency tests: deadlock guard, per-execution serialization, cross-execution parallelism.

Stack

This is PR 2 of 2. Builds on top of PR 1 (invoker state and checkpoint stamping). The diff here shows only the worker-model delta.

Closes #435

Replace the per-ARN lock with a per-execution worker that runs every
operation for one execution one at a time on its own serial lane.

- Worker model: SerialTaskLane, ExecutionWorker, ExecutionRegistry,
  ExecutionTask, CallableTask.
- One checkpoint write transaction in checkpoint/core.py shared by the
  web and in process paths.
- Checkpoint lifecycle actions returned as effects applied after the
  write. The reentrant notifier is gone.
- The invocation gate lives on the worker status.
- The blocking invoke runs off the lane so handler callbacks can run.
- Store integration test on memory, filesystem, and sqlite. Caught and
  fixed a non-atomic write in FileSystemStore.
- Runner accepts an optional store for store injection testing.
- docs/architecture.md: new, describes the worker model.

Closes #435
@yaythomas yaythomas force-pushed the feat/per-execution-worker branch from 5de9bf5 to b52ddaa Compare July 8, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant