Add per-worker rejects observability to the ckpool lane - #97
Conversation
Aggregate every Stratum submission by worker and reason (accepted, above_target, stale, duplicate, invalid_job, invalid_ntime, invalid_version, malformed) plus block-candidate submitblock outcomes, summing assigned difficulty per bucket, and write an atomic 60-second pool/rejects.status JSON for exporters. Purely additive observability patch on the pinned upstream ckpool: no changes to share validation, share responses, or block submission behavior. Ship live regression and load tests that drive a real patched binary over Stratum (make test-ckpool-rejects-observability, new CI job), fast patch-wiring and tamper-evidence tests, fake-RPC extensions for scripted submitblock results, and the exporter contract docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b97a5acd91
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Copy the pool tally and every worker tally under one rejects_lock hold so a rejects.status document can never show per-bucket worker sums exceeding pool totals while submissions are in flight (raised by Codex and Cursor Bugbot on #97). Worker pointers are collected first without the lock - ckpool never frees worker instances - and workers authorised mid-snapshot join the next cycle, so worker rows stay a subset of what the pool tally has seen. Also guard the block-outcome worker lookup against a NULL user. The path is unreachable today (get_user is get-or-create, matching upstream block_solve), but observability code must never be the thing that crashes the stratifier. The live suite now asserts snapshot consistency on every observed document including mid-load reads, and exact worker/pool equality once traffic settles; the README documents the guarantee. The Bugbot claim that the write interval follows update_interval is incorrect and left unchanged: statsupdate's cycle is hardcoded to 32 x 1.875s = 60s, and the live test measures ~60s write deltas with update_interval=30. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7da2b5fa00
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex flagged that the export walked every worker instance ever authorised, letting cheap authorization spam on the public Stratum port grow the status file, the snapshot allocations, and the copy performed under the submission-shared rejects_lock. Filter the snapshot to workers with at least one recorded submission or block outcome: the relaxed pre-filter read is safe because tallies only grow, and a worker racing its first submission simply joins the next 60s cycle. The live taxonomy test now authorises an idle worker and asserts it never appears in the export; the README documents the bound. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e831a5f000
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Track reject-observability workers in a lazy, lock-protected append-only registry so status snapshots avoid authorization scans and cannot drop active workers during count/fill races. Cover remote block outcomes and exercise real status replacement under concurrent traffic.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e62710a929
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Cap per-worker reject records at 4096, aggregate excess attributed events in a lock-consistent overflow tally, and expose truncation metadata. Add real-binary overflow coverage using a lowered test limit while preserving complete pool totals.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ddfee07b44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Expose the bounded registry setting in the ckpool service and example environment so operators can lower the compiled 4096 maximum. Pin the deployment wiring with the rejects observability contract tests.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dee3796. Configure here.
Remove the completed temporary file when atomic publication fails and pin both write- and rename-failure cleanup paths.
2 participants
CKPool now aggregates every Stratum submission by worker and reason (accepted, above_target, stale, duplicate, invalid_job, invalid_ntime, invalid_version, malformed) plus block-candidate submitblock outcomes, summing assigned difficulty per bucket, via a purely additive patch on the pinned upstream applied after the existing qbit patches. Tallies are written atomically (tmp+rename) to pool/rejects.status every 60 seconds as cumulative counters for exporters, with the contract documented in ckpool/README.md and cross-linked from docs/README.md; share validation, share responses, and block submission behavior are unchanged. A live regression/load suite drives a real patched binary over Stratum — covering the full reason taxonomy, both block outcomes, atomic-write parses under concurrent load, and exact per-worker count/difficulty conservation — via make test-ckpool-rejects-observability, a new scripts/build-patched-ckpool.sh, and a dedicated CI job. tests/fake_qbit_rpc.py gains additive knobs (scripted submitblock results, decoderawtransaction, a /control endpoint), and fast always-on tests pin the patch wiring, ordering, and tamper-evidence contract.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Note
Medium Risk
Touches hot-path share handling in ckpool's stratifier (locks and hooks on every submit) but is designed as read-only observability with bounded memory; regression coverage is strong, yet production mining behavior still depends on patch correctness under load.
Overview
Adds additive ckpool observability: every
mining.submitand localsubmitblockoutcome is tallied by worker and reason, then exported once per 60s stats cycle topool/rejects.statusvia tmp+rename. Share validation, Stratum responses, and block submission paths are not changed beyond recording counters after existing classification.The patched
stratifier.cdefines stable reason buckets (accepted,above_target,stale,duplicate,invalid_job,invalid_ntime,invalid_version,malformed) plusblock_accepted/block_rejected, with per-bucketcountand assigned-difficultydiff. Per-worker rows are lazy and capped (default 4096 viaQBIT_REJECTS_WORKER_LIMIT); overflow aggregates inworker_overflowwhen truncated.Build & deploy wiring:
qbit-rejects-observability.patchis applied after the existing qbit ckpool patches in the Docker image;compose.yamland.env.exampleforward the worker limit.scripts/build-patched-ckpool.shandmake test-ckpool-rejects-observabilitybuild a local binary for tests; CI adds a dedicated job.Tests & docs:
tests/test_ckpool_rejects_observability.pydrives real Stratum against a patched binary (full taxonomy, block outcomes, atomic writes under load);tests/test_ckpool_rejects_patch.pypins patch wiring.tests/fake_qbit_rpc.pygains scriptedsubmitblock, minimaldecoderawtransaction, and/control. Operator contract is documented inckpool/README.mdwith a cross-link fromdocs/README.md.Reviewed by Cursor Bugbot for commit 256116b. Bugbot is set up for automated code reviews on this repo. Configure here.