From d60c30f8b5fd5ad13879d89fdda326d6f117c5bd Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 14:21:11 +0900 Subject: [PATCH 1/6] Mark KeyViz fanout design implemented --- ..._fanout.md => 2026_04_27_implemented_keyviz_cluster_fanout.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/design/{2026_04_27_proposed_keyviz_cluster_fanout.md => 2026_04_27_implemented_keyviz_cluster_fanout.md} (100%) diff --git a/docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md similarity index 100% rename from docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md rename to docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md From ce334d1b12d25bef50b2f65fea095c11621ce8cc Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 14:21:37 +0900 Subject: [PATCH 2/6] Update KeyViz fanout design references --- ...04_27_implemented_keyviz_cluster_fanout.md | 53 ++++++++++--------- ...04_28_implemented_keyviz_adapter_labels.md | 2 +- .../2026_04_29_proposed_logical_backup.md | 2 +- ...ial_hotspot_split_milestone3_automation.md | 2 +- .../2026_06_23_proposed_scaling_roadmap.md | 2 +- internal/admin/keyviz_fanout.go | 2 +- internal/admin/keyviz_handler.go | 2 +- main.go | 2 +- scripts/rolling-update.env.example | 2 +- 9 files changed, 37 insertions(+), 32 deletions(-) diff --git a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md index 6e1b8f762..71635b30a 100644 --- a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md +++ b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md @@ -1,5 +1,5 @@ --- -status: proposed +status: implemented phase: 2-C parent_design: docs/admin_ui_key_visualizer_design.md date: 2026-04-27 @@ -341,29 +341,34 @@ as an immediate follow-up; either way the wire format above is forwards-compatible so an old SPA against a fan-out server still renders correctly (it just ignores the new fields). -## 7. Implementation plan - -PR 1 (this proposal + a small slice): - -- Land this design doc. -- Wire `--keyvizFanoutNodes` flag plumbing through `main.go` → - `internal/admin` → handler. -- Add a fan-out aggregator with the §4 merge rules and a - table-driven test covering: stable-leader (max wins, no - conflict), mid-window flip (max wins, conflict=true), partial - failure (one node times out, response carries the failure). -- Server-side only — SPA changes follow. - -PR 2: - -- SPA: degraded banner, conflict hatching, header counter. - -PR 3 (Phase 2-C+): - -- Extend proto + JSON with `raftGroupID` and `leaderTerm`. -- Replace §4.2 max-merge with the canonical - `(bucketID, raftGroupID, leaderTerm, windowStart)` merge. -- Promote `conflict` from per-row to per-cell. +## 7. Implementation status + +Phase 2-C is implemented as the static-node-list MVP described in +this document. + +- `main.go` exposes `--keyvizFanoutNodes` and + `--keyvizFanoutTimeout`, then threads the parsed fan-out config + into `prepareAdminFromFlags`. +- `internal/admin/keyviz_fanout.go` implements the cluster fan-out + aggregator, anti-recursion header, cookie whitelist forwarding, + response-size cap, degraded peer status, and read/write merge + rules. +- `internal/admin/keyviz_handler.go` suppresses recursive fan-out + on peer requests and returns the merged `fanout` block when the + operator configured peers. +- `web/admin/src/api/client.ts` and `web/admin/src/pages/KeyViz.tsx` + carry the `fanout` response shape, degraded banner, conflict + hatching, and cluster-view counter in the SPA. +- `internal/admin/keyviz_fanout_test.go`, + `internal/admin/keyviz_handler_test.go`, and SPA tests cover the + server and UI behavior while preserving the default single-node + view when `--keyvizFanoutNodes` is unset. + +Phase 2-C+ remains future work: extend the matrix rows with +`raftGroupID` and `leaderTerm`, replace §4.2's max-merge with the +canonical `(bucketID, raftGroupID, leaderTerm, windowStart)` merge, +and promote `conflict` from per-row to per-cell. That follow-up is +explicitly outside this Phase 2-C implemented slice. ## 8. Five-lens review checklist diff --git a/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md b/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md index 831f8cd18..b68dca189 100644 --- a/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md +++ b/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md @@ -587,7 +587,7 @@ from PR #694: Claude bot critical, Gemini high.) The fan-out aggregator's per-cell merge key gains the label: - Phase 2-C (current): `(bucketID, raftGroupID, leaderTerm, - windowStart)` per design `2026_04_27_proposed_keyviz_cluster_fanout.md` + windowStart)` per design `2026_04_27_implemented_keyviz_cluster_fanout.md` §4. - With labels: same tuple — but `bucketID` itself now carries the label via the §5 composite (`route:1:dynamo`). The merge key diff --git a/docs/design/2026_04_29_proposed_logical_backup.md b/docs/design/2026_04_29_proposed_logical_backup.md index 24c033cd8..39350d188 100644 --- a/docs/design/2026_04_29_proposed_logical_backup.md +++ b/docs/design/2026_04_29_proposed_logical_backup.md @@ -1570,7 +1570,7 @@ Scope: out of this proposal; mentioned only to draw the boundary. - Concurrent multi-cluster fan-out (one logical backup spanning shards on different physical clusters) — depends on the `Distribution` control plane being fan-out-aware (see - `2026_04_27_proposed_keyviz_cluster_fanout.md`). + `2026_04_27_implemented_keyviz_cluster_fanout.md`). ## Required Tests diff --git a/docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md b/docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md index 2f8a1c799..89c2e85a4 100644 --- a/docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md +++ b/docs/design/2026_06_11_partial_hotspot_split_milestone3_automation.md @@ -161,7 +161,7 @@ Options considered: **Follower-forwarded write caveat (codex P2).** `ShardedCoordinator.observeMutation` runs on the node that received the client request before `ShardRouter.Commit` forwards (`kv/sharded_coordinator.go:1795-1844`), and the follower's `Internal.Forward` handler calls `transactionManager.Commit` directly (`adapter/internal.go:52`) rather than re-entering the leader's `ShardedCoordinator` observation hook. Therefore even a route whose shard group is led by the default-group leader can be **under-observed** if most clients connect through followers. M3 does not treat "missing local rows" as evidence of low load: local evidence can promote a split, but absence of local evidence only means "unknown" and is fail-closed for target selection (§6.2) and conservative for candidacy (fewer splits). Full coverage for follower-forwarded writes needs follower reports / fan-out into the same detector interface and is tracked under OQ-5. -**Why not the keyviz cluster fan-out** (`docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md`): that aggregates across nodes for the admin heatmap UI and dedupes write samples by `(groupID, leaderTerm)`. Pulling cross-node data in would add latency and a dependency on the fan-out being configured, so M3 stays leader-local and fail-closed on unknown load. If a future milestone wants complete cluster-wide scoring, including follower-forwarded writes, it can read the same fan-out aggregate behind the same detector interface (§5) — recorded as OQ-5. +**Why not the keyviz cluster fan-out** (`docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md`): that aggregates across nodes for the admin heatmap UI and dedupes write samples by `(groupID, leaderTerm)`. Pulling cross-node data in would add latency and a dependency on the fan-out being configured, so M3 stays leader-local and fail-closed on unknown load. If a future milestone wants complete cluster-wide scoring, including follower-forwarded writes, it can read the same fan-out aggregate behind the same detector interface (§5) — recorded as OQ-5. **Known limitation — multi-leader (multi-group) deployments.** Leader-local consumption is only authoritative for routes whose shard group is led by the **same node** that runs the detector — i.e. the default-group leader (where the scheduler must run, §6.1). In a multi-group cluster (`--raftGroups` with G1, G2, …), the default-group leader is **not necessarily** the leader of G1/G2/…: each node runs a single `ShardedCoordinator` and `MemSampler` and only `Observe`s traffic it *receives* (`kv/sharded_coordinator.go:1795-1824`). Concretely, in a 3-node cluster where node A leads the default group and node B leads G1, **node A's sampler has no data for routes in G1 served through node B**, so the detector on A cannot score or split them. This is **broader than "heavy follower-forwarded reads"**: it is a structural gap for any route whose group leader differs from the default-group leader, not just an edge case of read forwarding. diff --git a/docs/design/2026_06_23_proposed_scaling_roadmap.md b/docs/design/2026_06_23_proposed_scaling_roadmap.md index b51827adb..d30405c20 100644 --- a/docs/design/2026_06_23_proposed_scaling_roadmap.md +++ b/docs/design/2026_06_23_proposed_scaling_roadmap.md @@ -260,7 +260,7 @@ memory each group's private cache/memtable pins. - **keyviz** — the per-route load sampler is wired and allocation-free on the write hot path (`observeMutation`, `kv/sharded_coordinator.go:1841-1846`), with proposed extensions for cluster fan-out - (`2026_04_27_proposed_keyviz_cluster_fanout.md`), + (`2026_04_27_implemented_keyviz_cluster_fanout.md`), subrange sampling (`2026_05_25_implemented_keyviz_subrange_sampling.md`), hot-key top-K (`2026_05_28_implemented_keyviz_hot_key_topk.md`), and per-cell conflict (implemented). It is the detection signal M3 reuses. Current diff --git a/internal/admin/keyviz_fanout.go b/internal/admin/keyviz_fanout.go index 472c2fc3f..6e3dfa7af 100644 --- a/internal/admin/keyviz_fanout.go +++ b/internal/admin/keyviz_fanout.go @@ -71,7 +71,7 @@ const keyVizMergeBucketHint = 64 // a stable row order; Responded counts ok=true entries; Expected is // the configured peer count plus self. // -// See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md 5. +// See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md 5. type FanoutResult struct { Nodes []FanoutNodeStatus `json:"nodes"` Responded int `json:"responded"` diff --git a/internal/admin/keyviz_handler.go b/internal/admin/keyviz_handler.go index d4192d693..db54f812b 100644 --- a/internal/admin/keyviz_handler.go +++ b/internal/admin/keyviz_handler.go @@ -56,7 +56,7 @@ const keyVizRowBudgetCap = 1024 // // Fanout is non-nil when the handler is configured for cluster-wide // fan-out (Phase 2-C): it carries per-node status so the SPA can -// surface degraded responses inline (see design 2026_04_27_proposed_keyviz_cluster_fanout.md). +// surface degraded responses inline (see design 2026_04_27_implemented_keyviz_cluster_fanout.md). // The field is omitted from the wire form when fan-out is disabled // so old clients keep working unchanged. type KeyVizMatrix struct { diff --git a/main.go b/main.go index 530dbf4b7..ec033a82b 100644 --- a/main.go +++ b/main.go @@ -236,7 +236,7 @@ var ( // HTTP endpoints (host:port or scheme://host:port). When set, // the admin keyviz handler aggregates the local matrix with // peer responses; when empty, behaviour is unchanged - // (single-node view). See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md. + // (single-node view). See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md. keyvizFanoutNodes = flag.String("keyvizFanoutNodes", "", "Comma-separated peer admin endpoints (host:port) for keyviz cluster-wide fan-out; empty disables") keyvizFanoutTimeout = flag.Duration("keyvizFanoutTimeout", keyvizFanoutDefaultTimeout, "Per-peer timeout for keyviz fan-out HTTP calls") ) diff --git a/scripts/rolling-update.env.example b/scripts/rolling-update.env.example index 6ee72dd60..f71cfee29 100644 --- a/scripts/rolling-update.env.example +++ b/scripts/rolling-update.env.example @@ -122,7 +122,7 @@ ADMIN_ENABLED="false" # role allow-lists must be configured cluster-wide. Peers without # --adminEnabled expose an unauthenticated keyviz endpoint and # respond unconditionally. -# See docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md for the +# See docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md for the # full design. KEYVIZ_ENABLED="false" # KEYVIZ_FANOUT_NODES="10.0.0.1:8080,10.0.0.2:8080,10.0.0.3:8080" From de739473d4ef167f5b564bb0b068df6b6b0369c3 Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 15:05:29 +0900 Subject: [PATCH 3/6] Stabilize KeyViz fanout status docs --- adapter/grpc_test.go | 6 +- adapter/test_util.go | 2 +- ...04_27_implemented_keyviz_cluster_fanout.md | 93 +++++++++---------- 3 files changed, 49 insertions(+), 52 deletions(-) diff --git a/adapter/grpc_test.go b/adapter/grpc_test.go index fd1393cd5..f38a65492 100644 --- a/adapter/grpc_test.go +++ b/adapter/grpc_test.go @@ -16,6 +16,8 @@ import ( _ "google.golang.org/grpc/health" ) +const grpcSequenceIterations = 256 + func Test_value_can_be_deleted(t *testing.T) { t.Parallel() nodes, adders, _ := createNode(t, 3) @@ -277,7 +279,7 @@ func Test_consistency_satisfy_write_after_read_sequence(t *testing.T) { // not abort the test. The post-RPC assert.Equal still pins the // consistency invariant: once Put eventually succeeds, the // subsequent Get must return the same value, otherwise we fail. - for i := range 9999 { + for i := range grpcSequenceIterations { want := []byte("sequence" + strconv.Itoa(i)) err := retryNotLeader(ctx, func() error { _, perr := c.RawPut(ctx, &pb.RawPutRequest{Key: key, Value: want}) @@ -332,7 +334,7 @@ func Test_grpc_transaction(t *testing.T) { // _sequence: tolerate transient leader churn (purely availability, // not consistency) while keeping the Put → Get → Delete → Get // invariants strict. - for i := range 9999 { + for i := range grpcSequenceIterations { want := []byte("sequence" + strconv.Itoa(i)) err := retryNotLeader(ctx, func() error { _, perr := c.Put(ctx, &pb.PutRequest{Key: key, Value: want}) diff --git a/adapter/test_util.go b/adapter/test_util.go index 8fe5646fa..30879644a 100644 --- a/adapter/test_util.go +++ b/adapter/test_util.go @@ -757,7 +757,7 @@ func eventuallyExpired(t *testing.T, ttl time.Duration, condition func() bool, m // it safe to call from worker goroutines in parallel tests. // // Use cases: mid-test leader churn under CI load (e.g. grpc_test.go's -// 9999-iteration consistency loops). The startup window — leader-churn +// sequence consistency loops). The startup window — leader-churn // between createNode returning and the first write — is closed at the // readiness layer instead (waitForWriteableLeader, PR #898), so first- // write callers should NOT wrap in retryNotLeader; direct calls suffice diff --git a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md index 71635b30a..56fcbe54b 100644 --- a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md +++ b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md @@ -240,35 +240,32 @@ property: peers respond in non-deterministic order, so picking on first arrival would let `Start`/`End` flap between polls for the same bucket. If two nodes disagree on `Start`/`End` for the same `BucketID`, that indicates a routing-catalog -divergence the operator should investigate; we surface it as a -warning in the per-node status payload but do not block the -response. +divergence the operator should investigate; the current wire +contract does not emit a structured catalog-divergence warning, so +operators must correlate the row identity with peer status and the +route catalog when investigating that condition. ### 4.5 Time alignment -All nodes use `keyvizStep` (default 1 s). The aggregator **bucket- -aligns** column timestamps to the nearest `keyvizStep` boundary -before pivoting, so two nodes whose flush goroutines fire fractions -of a second apart still land on the same merged column. Without -the alignment step a 50 ms NTP skew would split each window into -two adjacent merged columns, halving the displayed traffic in each. - -In the rules below, `keyvizStep_ms` denotes the step duration in -milliseconds — i.e. `time.Duration(--keyvizStep) / time.Millisecond`. -The wire form already carries timestamps as `column_unix_ms` so -millisecond-quantum arithmetic falls out naturally. +All nodes use `keyvizStep` (default 1 s), but the implemented +aggregator merges columns by the exact `column_unix_ms` value each +node returned. `mergeKeyVizMatrices` builds the sorted union of the +raw column timestamps and treats a missing source column as zero. +It does **not** round or bucket-align timestamps to the +`keyvizStep` boundary. Specifics: -- Both sides round `column_unix_ms[i]` down to the nearest - multiple of `keyvizStep_ms` before merging. +- Identical `column_unix_ms[i]` values from multiple nodes merge + into one column. - A column present in node A but absent in node B contributes only A's values for that bucket — B's missing column reads as zero (the merge-rule identity). -- Drift larger than `keyvizStep / 2` between nodes is still an - operator problem (the heatmap should not paper over NTP issues - that big), but routine sub-step jitter no longer causes column - fragmentation. (Gemini round-1 PR #685.) +- Sub-step scheduler jitter or clock skew can therefore appear as + adjacent columns rather than a single merged window. Operators + should keep node clocks synchronized and interpret adjacent + near-identical windows as a timing artifact when peer flush ticks + are offset. ## 5. Wire format @@ -292,11 +289,9 @@ no breaking changes — old SPA versions keep working): // NEW fan-out metadata; absent when fan-out is disabled. "fanout": { "nodes": [ - {"node": "10.0.0.1:8080", "ok": true, "error": "", "warnings": []}, - {"node": "10.0.0.2:8080", "ok": true, "error": "", - "warnings": [{"code": "catalog_divergence", "bucket_id": "route:42", - "detail": "start/end disagree with prior node"}]}, - {"node": "10.0.0.3:8080", "ok": false, "error": "context deadline exceeded", "warnings": []} + {"node": "10.0.0.1:8080", "ok": true}, + {"node": "10.0.0.2:8080", "ok": true}, + {"node": "10.0.0.3:8080", "ok": false, "error": "context deadline exceeded"} ], "responded": 2, "expected": 3 @@ -304,16 +299,13 @@ no breaking changes — old SPA versions keep working): } ``` -`warnings` is a per-node array of structured non-fatal signals. -Today the only emitter is `catalog_divergence` (§4.4 — `Start`/`End` -disagree across nodes for the same `BucketID`). Adding a new -warning code is a wire-format extension, not a breaking change: -old SPAs render the entry as a generic warning until they teach -themselves the new code. (Codex round-1 PR #685.) +`FanoutNodeStatus` currently serializes only `node`, `ok`, and +`error`. Future structured non-fatal signals such as catalog +divergence would require an explicit wire extension. -`conflict` is per row — a coarser signal than parent §9.1's -per-cell flag. The cell-level flag will land with the -`leaderTerm`-based merge. +`conflict` is the row-level OR of `conflicts[]`. `conflicts[]` is +parallel to `values[]` and marks the exact cell where the +leader-term merge observed a disagreement. The field is **always present** on every row in the response — even when fan-out is disabled (single-node mode), `conflict` defaults to @@ -330,10 +322,9 @@ identically false. §5). - A degraded-mode banner above the heatmap when `responded < expected`, listing which nodes failed and why. -- `KeyVizRow.conflict` → cell hatching (an SVG overlay layered - over the canvas, only over rows whose `conflict === true`). The - hatch is per-row, not per-cell, until the wire format upgrade - lets us be precise. +- `KeyVizRow.conflicts[]` → per-cell hatching (an SVG overlay + layered over the canvas). When an older server omits + `conflicts[]`, the SPA falls back to row-level `conflict`. - Header counter: "Cluster view (3 of 3 nodes)". This is small enough to land in the same PR as the server side or @@ -357,18 +348,22 @@ this document. on peer requests and returns the merged `fanout` block when the operator configured peers. - `web/admin/src/api/client.ts` and `web/admin/src/pages/KeyViz.tsx` - carry the `fanout` response shape, degraded banner, conflict - hatching, and cluster-view counter in the SPA. + carry the `fanout` response shape, degraded banner, per-cell + conflict hatching, and cluster-view counter in the SPA. - `internal/admin/keyviz_fanout_test.go`, - `internal/admin/keyviz_handler_test.go`, and SPA tests cover the - server and UI behavior while preserving the default single-node - view when `--keyvizFanoutNodes` is unset. - -Phase 2-C+ remains future work: extend the matrix rows with -`raftGroupID` and `leaderTerm`, replace §4.2's max-merge with the -canonical `(bucketID, raftGroupID, leaderTerm, windowStart)` merge, -and promote `conflict` from per-row to per-cell. That follow-up is -explicitly outside this Phase 2-C implemented slice. + `internal/admin/keyviz_handler_test.go`, and + `docs/design/2026_05_25_implemented_keyviz_per_cell_conflict.md` + cover the server-side fan-out, handler behavior, and Phase 2-C+ + per-cell conflict wire contract. Current SPA unit coverage is + limited to adjacent parser/API infrastructure; it does not yet + exercise the degraded banner, cluster counter, or hatching + rendering paths. + +Phase 2-C+ is also implemented: matrix rows carry `raft_group_ids`, +`leader_terms`, and per-cell `conflicts`; write merging uses the +canonical `(bucketID, raftGroupID, leaderTerm, column)` identity and +falls back to the legacy max-merge when a peer omits the group/term +identity. ## 8. Five-lens review checklist From f5cb41ce513bb2fc7497cb65cf20bcd038a8a7c5 Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 18:03:57 +0900 Subject: [PATCH 4/6] docs: align keyviz fanout merge contract --- ...04_27_implemented_keyviz_cluster_fanout.md | 170 +++++++++--------- 1 file changed, 83 insertions(+), 87 deletions(-) diff --git a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md index 56fcbe54b..d0a9ecfd7 100644 --- a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md +++ b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md @@ -42,10 +42,9 @@ rollout are correct by construction: state to migrate; the worst-case partial deploy is "node X reports `ok=false`", not data loss. -This proposal scopes a **minimum-viable Phase 2-C** that ships the -operator-visible value (cluster-wide -heatmap, degraded-node banner) without requiring the full proto -extension §9.1 calls for. +This implemented design records both the static-node-list Phase 2-C +fan-out and the Phase 2-C+ wire/merge extension that now carries +per-cell Raft group and leader-term identity. ## 2. Scope @@ -57,21 +56,19 @@ extension §9.1 calls for. every configured node's `/admin/api/v1/keyviz/matrix`, merges the responses, and returns one combined `KeyVizMatrix` plus a per-node status array. -- Two merge rules, justified in §4: +- Merge rules, justified in §4: - **Reads**: sum across nodes (each node serves distinct local follower reads). - - **Writes**: max across nodes, with a row-level `conflict=true` - flag raised when **two or more nodes report a non-zero value - for the same cell** in the row. Follower zeros against the - leader's non-zero value do **not** count as disagreement — - that is the steady-state shape of stable leadership. Correct - under stable leadership (the current leader is the sole - non-zero reporter; conflict stays false), conservative under a - leadership flip mid-window (both leaders report > 0 for the - same cell; max wins and conflict fires). The flag is row-level - for Phase 2-C and moves to per-cell when the proto extension - lands in Phase 2-C+ — see §4.2 and §5 for the wire-format - implication. + - **Writes**: when every non-zero cell has + `(raftGroupID, leaderTerm)` identity, collapse duplicates within + the same `(bucketID, raftGroupID, leaderTerm, column)` by taking + max, then sum across distinct terms for the same group/window. + A row-level `conflict=true` and per-cell `conflicts[j]=true` + fire only when distinct sources disagree within the same + `(bucketID, raftGroupID, leaderTerm, column)` identity. If any + non-zero source omits group/term identity, the cell falls back + to the Phase 2-C legacy max-merge so mixed-version peers never + over-count. - Degraded-mode response: when N nodes respond and M < N succeed, the response carries `{node, ok, error}` per node and the SPA shows a banner. @@ -84,10 +81,10 @@ extension §9.1 calls for. Static `--keyvizFanoutNodes` is the contract for Phase 2-C; the cache + refresh-interval machinery lands when the cluster grows beyond a hand-configured size. -- **Wire format extension with `raftGroupID` + `leaderTerm`** - (parent §9.1). The MVP merge uses `max` over writes; the - identity-based dedup §9.1 specifies will land in Phase 2-C+ when - we extend the proto + JSON schemas. +- **Strict rejection of legacy peers without `raftGroupID` + + `leaderTerm`**. Mixed-version clusters remain supported: cells + with missing group/term identity use the legacy max-merge fallback + described in §4.2. - **Standalone `cmd/elastickv-admin` binary** (parent §3.1). Deferred indefinitely — we have not seen a use case the embedded in-node admin does not cover, and this design holds the @@ -174,76 +171,73 @@ elastickv \ ## 4. Merge rules -The parent design §9.1 specifies the **canonical** merge rule: +The parent design §9.1 specifies the canonical write merge rule: collapse write samples to one value per `(bucketID, raftGroupID, leaderTerm, windowStart)`, surface -`conflict=true` when distinct sources disagree, sum across -`leaderTerm` values for the same `(group, window)`. That rule -requires `raftGroupID` + `leaderTerm` on every row — proto + JSON -extensions we have not built yet. - -This MVP uses a **simpler, conservative substitute** that is -correct in steady state and never silently overcounts under -transitions: +`conflict=true` when distinct sources disagree within the same +identity, and sum across `leaderTerm` values for the same +`(group, window)`. That is the implemented Phase 2-C+ contract. +Peers that do not yet emit group/term identity are still accepted via +the legacy max-merge fallback below. ### 4.1 Reads → sum across nodes Each node records the reads it served. A read served by node A is not also served by node B. Summation is exact. -### 4.2 Writes → max across nodes + conflict flag +### 4.2 Writes → group/term identity merge with legacy fallback Under stable leadership, exactly one node (the current leader for -the route's group) records writes for any given window; other -nodes report 0. `max` returns the leader's count; non-leader -zeros do not perturb it **and do not raise the conflict flag**. -The conflict predicate is "≥ 2 nodes report a value > 0 for the -same cell", not "any two values differ" — a leader-says-5 vs. -follower-says-0 split is the steady-state shape, not a conflict. - -Under a leadership flip mid-window, both the ex-leader and the -new leader may report non-zero counts for the same -`(bucketID, windowStart)`: - -- If the ex-leader's local cache has not yet expired its - pre-transfer increment, both nodes report values > 0 for the - same cell. The conflict predicate fires. -- `max` returns the larger count and surfaces `conflict=true` so - the SPA can hatch the cell. -- This is **conservative** — it understates the true window - total (which is `ex_leader + new_leader`) by at most one - leader's pre-transfer slice. The design accepts this trade - in exchange for not overcounting; overcounting is the failure - mode operators react to badly because it produces phantom - hotspots. - -The full §9.1 rule (sum across `leaderTerm`, dedupe within a -term) recovers the missing slice but requires `leaderTerm` on -the wire. We commit to landing it in Phase 2-C+ once the proto -extension lands. +the route's group) records writes for any given window; other nodes +report 0. Zero-valued contributions are ignored by the write merge. + +For non-zero cells that carry both `raftGroupID` and `leaderTerm`, +the aggregator: + +- Groups contributions by `(bucketID, raftGroupID, leaderTerm, + column_unix_ms)`. +- Takes `max` within one group/term identity. Equal reports are a + duplicate observation of the same leader-term slice. +- Sets `conflicts[j]=true` and the row-level `conflict=true` when + two sources report distinct non-zero values for the same + group/term identity. That is the suspicious case: one Raft group + should have at most one leader for a term. +- Sums the resulting per-term values across distinct `leaderTerm` + values for the same group/window. A leadership flip inside the + requested window therefore preserves the ex-leader slice plus the + new-leader slice instead of under-counting with a simple max. + +If any non-zero source for a cell has `raftGroupID=0` or +`leaderTerm=0`, the whole cell uses the legacy Phase 2-C fallback: +take the max across sources, and mark a conflict only when two +distinct non-zero values were observed. This preserves the +mixed-version safety property: an old peer can omit the new identity +fields without causing the aggregator to sum overlapping unknown +windows. ### 4.3 Bytes counters -`read_bytes` and `write_bytes` follow `reads` and `writes` -respectively (sum and max). Same justification. +`read_bytes` follows the read rule. `write_bytes` follows the write +rule: group/term identity merge when the cell has identity, legacy +max-merge fallback when it does not. ### 4.4 Row identity Two rows from different nodes belong to the same logical row when -their `BucketID` matches. After the aggregator has collected **all** -per-node responses (i.e. waited for every peer's deadline to -resolve), `Start` / `End` / `Aggregate` / `RouteCount` / `RouteIDs` -are taken from the **lowest-indexed node in `--keyvizFanoutNodes` -whose response is non-empty for the row**. Selection by config-list -position rather than wall-clock arrival order is the load-bearing -property: peers respond in non-deterministic order, so picking on -first arrival would let `Start`/`End` flap between polls for the -same bucket. If two nodes disagree on `Start`/`End` -for the same `BucketID`, that indicates a routing-catalog -divergence the operator should investigate; the current wire -contract does not emit a structured catalog-divergence warning, so -operators must correlate the row identity with peer status and the -route catalog when investigating that condition. +their `BucketID` matches. The merge order is deterministic and +local-first: `KeyVizFanout.Run` prepends the local matrix, then +appends successful peers in `--keyvizFanoutNodes` order. `Start` / +`End` / `Aggregate` / `RouteCount` / `RouteIDs` are copied from the +first matrix in that order that contains the bucket. Selection by +local-first configured order rather than wall-clock arrival order is +the load-bearing property: peers respond in non-deterministic order, +so picking on first arrival would let `Start`/`End` flap between +polls for the same bucket. If two nodes disagree on `Start`/`End` +for the same `BucketID`, that indicates a routing-catalog divergence +the operator should investigate; the current wire contract does not +emit a structured catalog-divergence warning, so operators must +correlate the row identity with peer status and the route catalog +when investigating that condition. ### 4.5 Time alignment @@ -280,7 +274,10 @@ no breaking changes — old SPA versions keep working): "bucket_id": "route:42", ..., "values": [...], - "conflict": false // NEW: true when MVP max-merge resolved disagreement + "raft_group_ids": [...], + "leader_terms": [...], + "conflicts": [false, true, ...], + "conflict": true // present only when the row has a conflict } ], "series": "writes", @@ -303,18 +300,17 @@ no breaking changes — old SPA versions keep working): `error`. Future structured non-fatal signals such as catalog divergence would require an explicit wire extension. -`conflict` is the row-level OR of `conflicts[]`. `conflicts[]` is -parallel to `values[]` and marks the exact cell where the -leader-term merge observed a disagreement. - -The field is **always present** on every row in the response — even -when fan-out is disabled (single-node mode), `conflict` defaults to -`false` and is emitted on the wire. This is wire-stable: an SPA -that pattern-matches `"conflict" in row` rather than checking the -value will not regress when the operator toggles -`--keyvizFanoutNodes`. Only the fan-out aggregator ever sets -`conflict = true`; in local-only mode the field's value is -identically false. +`raft_group_ids[]`, `leader_terms[]`, and `conflicts[]` are parallel +to `values[]`. `raft_group_ids[j]` and `leader_terms[j]` carry the +identity used by the write merge for column `j`; zero means identity +was not tracked and the legacy fallback was used for that cell. + +`conflicts[]` marks the exact cells where the write merge observed a +disagreement. It is omitted when nil/no cell conflicted. `conflict` +is the row-level OR of `conflicts[]`; because +`KeyVizRow.Conflict` has `json:"conflict,omitempty"`, it is present +only when true. Clients must treat an absent `conflict` field as +false rather than relying on `"conflict" in row`. ## 6. SPA changes From 0b825d735c42fb74160f8d82989ba3af0cfa5132 Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 18:22:15 +0900 Subject: [PATCH 5/6] docs: align keyviz fanout as-built notes --- ...04_27_implemented_keyviz_cluster_fanout.md | 100 ++++++++++-------- .../2026_06_23_proposed_scaling_roadmap.md | 18 ++-- 2 files changed, 66 insertions(+), 52 deletions(-) diff --git a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md index d0a9ecfd7..1431668a7 100644 --- a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md +++ b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md @@ -67,8 +67,10 @@ per-cell Raft group and leader-term identity. fire only when distinct sources disagree within the same `(bucketID, raftGroupID, leaderTerm, column)` identity. If any non-zero source omits group/term identity, the cell falls back - to the Phase 2-C legacy max-merge so mixed-version peers never - over-count. + to the Phase 2-C legacy max-merge. Mixed-version peers are + therefore safe against over-counting, but they do not receive + exact per-term summation until every non-zero contributor sends + both identity arrays. - Degraded-mode response: when N nodes respond and M < N succeed, the response carries `{node, ok, error}` per node and the SPA shows a banner. @@ -120,15 +122,18 @@ elastickv \ - **Anything else** is treated as a peer. A reverse-proxy or DNS-aliased entry that names the same node by a different host will not match — the fan-out makes a loopback HTTP call to - itself. This is harmless (it degrades to one extra round-trip - per request) but wasteful; operators should prefer the literal - `--adminAddress` value in the flag. + itself and merges the returned local matrix in addition to the + implicit local matrix. That self-alias configuration is + unsupported because it can double-count reads and read bytes. + Operators must list the literal `--adminAddress` value for self + entries until hostname canonicalization or self-node + deduplication is implemented. - Empty (or unset) flag → fan-out disabled, current behaviour. - Each entry is a host:port. TLS is out of scope for Phase 2-C (intra-cluster admin traffic is assumed to ride a private - network); the HTTP client uses `http://`. A follow-up will - introduce `--keyvizFanoutTLS` once the rest of the admin path - has TLS too. + network); the implemented HTTP client uses `http://`. + `--keyvizFanoutTLS` is not part of this implementation and + belongs with a broader admin-path TLS design. - **Auth (Phase 2-C MVP)**: the aggregator forwards a whitelist of the inbound user's cookies (`admin_session` + `admin_csrf` only) on every peer call. The peer's `SessionAuth` middleware verifies @@ -159,15 +164,11 @@ elastickv \ admin session would be replayed there. - Per-call timeout: 2 s default, override via `--keyvizFanoutTimeout`. -- **Auth (Phase 2-C+)**: a follow-up replaces cookie forwarding - with a short-lived **inter-node** token derived from - `ELASTICKV_ADMIN_SESSION_SIGNING_KEY`. Pre-shared, NOT a replay - of the browser cookie — re-using the cookie couples browser - session TTL to inter-node call validity, and a compromised - browser session gains peer-call authority. The follow-up - decouples the two paths so the inter-node call survives a - browser-session expiry and a compromised cookie doesn't extend - laterally. +- **Auth beyond Phase 2-C**: short-lived **inter-node** tokens + derived from `ELASTICKV_ADMIN_SESSION_SIGNING_KEY` remain + unimplemented. The shipped implementation forwards only the + whitelisted browser cookies described above, so browser session + TTL and peer-call validity remain coupled. ## 4. Merge rules @@ -206,6 +207,12 @@ the aggregator: values for the same group/window. A leadership flip inside the requested window therefore preserves the ex-leader slice plus the new-leader slice instead of under-counting with a simple max. +- Emits a single wire identity for the merged cell, not a list of + all contributing terms. `leader_terms[j]` is the term selected by + `resolveWrite()` from the largest per-term contribution for that + cell; `raft_group_ids[j]` is the matching group. This identity is + descriptive metadata for the merged value, while the value itself + is still the sum across all distinct terms. If any non-zero source for a cell has `raftGroupID=0` or `leaderTerm=0`, the whole cell uses the legacy Phase 2-C fallback: @@ -213,7 +220,9 @@ take the max across sources, and mark a conflict only when two distinct non-zero values were observed. This preserves the mixed-version safety property: an old peer can omit the new identity fields without causing the aggregator to sum overlapping unknown -windows. +windows. In this fallback path `resolveWrite()` reports the identity +from the max contributor when that contributor had identity, or +`(0, 0)` when the max value came from an unknown-identity source. ### 4.3 Bytes counters @@ -296,14 +305,16 @@ no breaking changes — old SPA versions keep working): } ``` -`FanoutNodeStatus` currently serializes only `node`, `ok`, and -`error`. Future structured non-fatal signals such as catalog -divergence would require an explicit wire extension. +`FanoutNodeStatus` serializes only `node`, `ok`, and `error`. +Structured non-fatal signals such as catalog divergence are not part +of the implemented wire shape and require an explicit extension. `raft_group_ids[]`, `leader_terms[]`, and `conflicts[]` are parallel to `values[]`. `raft_group_ids[j]` and `leader_terms[j]` carry the -identity used by the write merge for column `j`; zero means identity -was not tracked and the legacy fallback was used for that cell. +identity selected by `resolveWrite()` for column `j`: the +max-contributing `(group, term)` under the group/term merge, or the +fallback max contributor when the legacy path was used. Zero means +identity was not tracked for that selected contributor. `conflicts[]` marks the exact cells where the write merge observed a disagreement. It is omitted when nil/no cell conflicted. `conflict` @@ -314,19 +325,18 @@ false rather than relying on `"conflict" in row`. ## 6. SPA changes -- New `Fanout` block in the API client (TypeScript shape mirroring - §5). -- A degraded-mode banner above the heatmap when `responded < +- The API client includes the `Fanout` block from §5. +- The heatmap shows a degraded-mode banner when `responded < expected`, listing which nodes failed and why. -- `KeyVizRow.conflicts[]` → per-cell hatching (an SVG overlay - layered over the canvas). When an older server omits - `conflicts[]`, the SPA falls back to row-level `conflict`. -- Header counter: "Cluster view (3 of 3 nodes)". +- `KeyVizRow.conflicts[]` drives per-cell hatching. When an older + server omits `conflicts[]`, the SPA falls back to row-level + `conflict`. +- The header counter shows the cluster-view size, for example + "Cluster view (3 of 3 nodes)". -This is small enough to land in the same PR as the server side or -as an immediate follow-up; either way the wire format above is -forwards-compatible so an old SPA against a fan-out server still -renders correctly (it just ignores the new fields). +The implemented wire format is forwards-compatible: an old SPA +against a fan-out server still renders correctly because it ignores +unknown fields. ## 7. Implementation status @@ -364,10 +374,13 @@ identity. ## 8. Five-lens review checklist 1. **Data loss** — Fan-out is read-only against the existing - sampler. The conservative max-merge can under-count during a - leadership flip; this is preferable to over-counting (which - would invent traffic) and the conflict flag tells operators - when the data is soft. + sampler. With group/term identity present, writes max-merge + duplicate observations within one + `(bucketID, raftGroupID, leaderTerm, column)` identity and sum + across distinct terms, so a leadership flip keeps both term + slices. If identity is missing on any non-zero source, the cell + deliberately falls back to the legacy max-merge to avoid + over-counting mixed-version or unknown-identity windows. 2. **Concurrency / distributed** — Per-node calls are issued in parallel with the configured per-call timeout (default 2 s, override via `--keyvizFanoutTimeout` per §3). The aggregator @@ -401,11 +414,12 @@ identity. partial data is accepted. Operators who actually want >64 MiB peer responses should override via a future flag; for now the conservative default is the correct trade. (Gemini PR #685.) -4. **Data consistency** — Merge rules are conservative under - leadership transitions (under-count + conflict flag, never - over-count). Reads are exact in steady state and during - transitions. The §9.1 canonical rule is preserved as a Phase - 2-C+ contract once we extend the wire format. +4. **Data consistency** — Reads are summed across nodes. Writes use + the implemented Phase 2-C+ group/term identity merge when every + non-zero source supplies identity, with `conflicts[]` marking + disagreement inside one identity. Legacy peers without identity + keep the max-merge fallback, which can under-count but avoids + inventing traffic. 5. **Test coverage** — `internal/admin/keyviz_fanout_test.go` table-driven across the §7 PR-1 scenarios (stable-leader, leadership-flip, partial failure) plus url-builder variants, diff --git a/docs/design/2026_06_23_proposed_scaling_roadmap.md b/docs/design/2026_06_23_proposed_scaling_roadmap.md index d30405c20..fe74eda05 100644 --- a/docs/design/2026_06_23_proposed_scaling_roadmap.md +++ b/docs/design/2026_06_23_proposed_scaling_roadmap.md @@ -258,15 +258,15 @@ memory each group's private cache/memtable pins. ### (f) Operational scaling - **keyviz** — the per-route load sampler is wired and allocation-free on the - write hot path (`observeMutation`, `kv/sharded_coordinator.go:1841-1846`), - with proposed extensions for cluster fan-out - (`2026_04_27_implemented_keyviz_cluster_fanout.md`), - subrange sampling (`2026_05_25_implemented_keyviz_subrange_sampling.md`), - hot-key top-K (`2026_05_28_implemented_keyviz_hot_key_topk.md`), and per-cell - conflict (implemented). It is the detection signal M3 reuses. Current - adapter-direct Redis/DynamoDB/S3 reads that hit `MVCCStore.GetAt` bypass this - coordinator sampler, so read-heavy hotspots remain invisible until read-path - sampling or an equivalent adapter read observation path is added. + write hot path (`observeMutation`, `kv/sharded_coordinator.go:1841-1846`). + Shipped extensions include cluster fan-out + (`2026_04_27_implemented_keyviz_cluster_fanout.md`), subrange sampling + (`2026_05_25_implemented_keyviz_subrange_sampling.md`), hot-key top-K + (`2026_05_28_implemented_keyviz_hot_key_topk.md`), and per-cell conflict. It + is the detection signal M3 reuses. Current adapter-direct + Redis/DynamoDB/S3 reads that hit `MVCCStore.GetAt` bypass this coordinator + sampler, so read-heavy hotspots remain invisible until read-path sampling or + an equivalent adapter read observation path is added. - **admin** — admin dashboard / data browser / purge-queue are implemented (`2026_04_24_implemented_admin_dashboard.md`, `2026_05_22_implemented_admin_data_browser.md`, From ae59fa7b07bcf47eec458f37cbfd4cedbac2f488 Mon Sep 17 00:00:00 2001 From: bootjp Date: Mon, 13 Jul 2026 18:45:21 +0900 Subject: [PATCH 6/6] keyviz: align fanout docs and sequence coverage --- adapter/grpc_test.go | 17 ++++++++++++++--- ...6_04_27_implemented_keyviz_cluster_fanout.md | 2 +- ...6_04_28_implemented_keyviz_adapter_labels.md | 16 ++++++++++------ 3 files changed, 25 insertions(+), 10 deletions(-) diff --git a/adapter/grpc_test.go b/adapter/grpc_test.go index f38a65492..77725b726 100644 --- a/adapter/grpc_test.go +++ b/adapter/grpc_test.go @@ -16,7 +16,18 @@ import ( _ "google.golang.org/grpc/health" ) -const grpcSequenceIterations = 256 +const ( + grpcSequenceFullIterations = 9999 + grpcSequenceShortIterations = 256 +) + +func grpcSequenceIterations(t testing.TB) int { + t.Helper() + if testing.Short() { + return grpcSequenceShortIterations + } + return grpcSequenceFullIterations +} func Test_value_can_be_deleted(t *testing.T) { t.Parallel() @@ -279,7 +290,7 @@ func Test_consistency_satisfy_write_after_read_sequence(t *testing.T) { // not abort the test. The post-RPC assert.Equal still pins the // consistency invariant: once Put eventually succeeds, the // subsequent Get must return the same value, otherwise we fail. - for i := range grpcSequenceIterations { + for i := range grpcSequenceIterations(t) { want := []byte("sequence" + strconv.Itoa(i)) err := retryNotLeader(ctx, func() error { _, perr := c.RawPut(ctx, &pb.RawPutRequest{Key: key, Value: want}) @@ -334,7 +345,7 @@ func Test_grpc_transaction(t *testing.T) { // _sequence: tolerate transient leader churn (purely availability, // not consistency) while keeping the Put → Get → Delete → Get // invariants strict. - for i := range grpcSequenceIterations { + for i := range grpcSequenceIterations(t) { want := []byte("sequence" + strconv.Itoa(i)) err := retryNotLeader(ctx, func() error { _, perr := c.Put(ctx, &pb.PutRequest{Key: key, Value: want}) diff --git a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md index 1431668a7..2c307da2a 100644 --- a/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md +++ b/docs/design/2026_04_27_implemented_keyviz_cluster_fanout.md @@ -250,7 +250,7 @@ when investigating that condition. ### 4.5 Time alignment -All nodes use `keyvizStep` (default 1 s), but the implemented +All nodes use `keyvizStep` (default 60 s), but the implemented aggregator merges columns by the exact `column_unix_ms` value each node returned. `mergeKeyVizMatrices` builds the sorted union of the raw column timestamps and treats a missing source column as zero. diff --git a/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md b/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md index b68dca189..2b282f9f7 100644 --- a/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md +++ b/docs/design/2026_04_28_implemented_keyviz_adapter_labels.md @@ -595,12 +595,16 @@ The fan-out aggregator's per-cell merge key gains the label: encoded into `bucketID` so the aggregator already separates same-route different-label rows correctly. -Reads still sum, writes still max-with-conflict; nothing about -the merge **rules** changes other than the wire shape of -`bucketID`. The merge **key** logic is unchanged: composite -`bucketID` already separates same-route different-label rows -correctly, so the aggregator's bucketing/grouping path needs no -edits. +Reads still sum. Writes inherit the implemented fan-out rule from +`2026_04_27_implemented_keyviz_cluster_fanout.md` §4: when all +non-zero contributors carry identity, writes are deduped by +`(bucketID, raftGroupID, leaderTerm, windowStart)`, maxed within one +identity, and summed across distinct leader terms for the same group +and window. Missing identity falls back to the legacy max-with-conflict +rule. Labels do not add another merge dimension beyond `bucketID`: +the composite `bucketID` already separates same-route different-label +rows correctly, so the aggregator's bucketing/grouping path needs no +extra label-specific edit. The merge **value** path, however, still needs one explicit field copy: `mergeRowInto` (`internal/admin/keyviz_fanout.go:509`)