feat(testing): per-execution worker serialization#516
Draft
yaythomas wants to merge 1 commit into
Draft
Conversation
99be1b4 to
5017a88
Compare
c39656a to
1d3d835
Compare
5017a88 to
11eebdc
Compare
1d3d835 to
5de9bf5
Compare
11eebdc to
8a6058c
Compare
Replace the per-ARN lock with a per-execution worker that runs every operation for one execution one at a time on its own serial lane. - Worker model: SerialTaskLane, ExecutionWorker, ExecutionRegistry, ExecutionTask, CallableTask. - One checkpoint write transaction in checkpoint/core.py shared by the web and in process paths. - Checkpoint lifecycle actions returned as effects applied after the write. The reentrant notifier is gone. - The invocation gate lives on the worker status. - The blocking invoke runs off the lane so handler callbacks can run. - Store integration test on memory, filesystem, and sqlite. Caught and fixed a non-atomic write in FileSystemStore. - Runner accepts an optional store for store injection testing. - docs/architecture.md: new, describes the worker model. Closes #435
5de9bf5 to
b52ddaa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the per-ARN lock with a per-execution worker that runs every operation for one execution one at a time on its own serial lane. Serialization and completion ordering are now structural properties of the design.
Changes
Worker model (
worker/package):SerialTaskLane: single-consumer FIFO queue with one daemon thread. Tasks run in submit order with no overlap.ExecutionWorker: runs every operation for one execution on its lane. Carries invocation state from each task into the next. Tears down on terminal status.ExecutionRegistry: owns one worker per execution ARN. Creates on first use, drops on completion.ExecutionTask/TaskOutcome: contract for work run on a worker. Exclusive access to execution state for the duration of execute.CallableTask: lets existing methods serialize on the lane without restating as a task.Shared checkpoint write-transaction (
checkpoint/core.py):CheckpointCore.applyis the single write-transaction shared by the web and in-process paths (extendsmatch_cachedfrom PR 1).checkpoint/effects.py) applied after the write. The reentrant notifier is removed.Invocation runs off the lane:
Store coverage:
storeparameter for store-injection testing.FileSystemExecutionStorewrites saves atomically (temp file + replace).tests/e2e/store_backed_lifecycle_test.pydrives a full lifecycle on memory, filesystem, and sqlite.Documentation:
docs/architecture.md: new, describes the worker model, checkpoint flow, invocation state machine, and pagination.Testing
Stack
This is PR 2 of 2. Builds on top of PR 1 (invoker state and checkpoint stamping). The diff here shows only the worker-model delta.
Closes #435