feat: implement background DEK re-encryption pipeline

[gtema]

Replaces the log-only stub with a real background task: on each DEK
rotation, sweep every not-yet-fully-migrated retired epoch and
re-encrypt its remaining records under the current epoch, using an
optimistic CAS retry (skip after 3 losses, retried on the next
rotation cycle) since Fjall's plain Keyspace/Batch API has no
built-in compare-and-swap (ADR 0016-v2 §6 step 5 / §6.2 step 4).
Fully migrated epochs are marked done so they aren't rescanned by
future rotations.

Signed-off-by: Artem Goncharov artem.goncharov@gmail.com

[github-actions]

🦢 Load Test Results

Goose Attack Report

Plan Overview

Action	Started	Stopped	Elapsed	Users
Increasing	26-07-02 10:11:31	26-07-02 10:11:41	00:00:10	0 → 20
Maintaining	26-07-02 10:11:41	26-07-02 10:12:12	00:00:31	20
Decreasing	26-07-02 10:12:12	26-07-02 10:12:12	00:00:00	0 ← 20

Request Metrics

Method	Name	# Requests	# Fails	Average (ms)	Min (ms)	Max (ms)	RPS	Failures/s
DELETE	DELETE /v3/auth/tokens	842	0	64.71	9	100	28.07	0.00
DELETE	DELETE /v3/projects/:id (teardown)	2	0	39.50	37	42	0.07	0.00
DELETE	DELETE /v3/users/:id (teardown)	3	0	26.67	25	30	0.10	0.00
GET		4944	0	60.43	41	116	164.80	0.00
GET	GET /v3/auth/tokens (validate new)	839	0	64.57	36	84	27.97	0.00
GET	GET /v3/projects/:id	1200	0	49.67	42	67	40.00	0.00
GET	GET /v3/users/:id	1675	0	53.41	45	73	55.83	0.00
POST	POST /v3/auth/tokens	838	0	48.67	37	64	27.93	0.00
	Aggregated	10343	0	57.76	9	116	344.77	0.00

Response Time Metrics

Method	Name	50%ile (ms)	60%ile (ms)	70%ile (ms)	80%ile (ms)	90%ile (ms)	95%ile (ms)	99%ile (ms)	100%ile (ms)
DELETE	DELETE /v3/auth/tokens	64	65	66	67	69	71	79	100
DELETE	DELETE /v3/projects/:id (teardown)	37	37	37	42	42	42	42	42
DELETE	DELETE /v3/users/:id (teardown)	25	25	25	25	30	30	30	30
GET		53	57	59	67	93	96	100	116
GET	GET /v3/auth/tokens (validate new)	64	65	66	67	69	71	79	84
GET	GET /v3/projects/:id	49	50	51	51	53	55	61	67
GET	GET /v3/users/:id	53	54	54	55	57	59	66	73
POST	POST /v3/auth/tokens	48	49	50	51	52	54	60	64
	Aggregated	53	56	60	64	71	93	99	116

Status Code Metrics

Method	Name	Status Codes
DELETE	DELETE /v3/auth/tokens	842 [204]
DELETE	DELETE /v3/projects/:id (teardown)	2 [204]
DELETE	DELETE /v3/users/:id (teardown)	3 [204]
GET		4,944 [200]
GET	GET /v3/auth/tokens (validate new)	839 [200]
GET	GET /v3/projects/:id	1,200 [200]
GET	GET /v3/users/:id	1,675 [200]
POST	POST /v3/auth/tokens	838 [200]
	Aggregated	847 [204], 9,496 [200]

Transaction Metrics

Transaction	# Times Run	# Fails	Average (ms)	Min (ms)	Max (ms)	RPS	Failures/s
ReadHeavy
0.0	0	0	0.00	0	0	0.00	0.00
0.1	1328	0	58.65	51	80	44.27	0.00
0.2	1329	0	49.23	42	66	44.30	0.00
0.3	1329	0	49.32	41	67	44.30	0.00
TokenLifecycle
1.0	0	0	0.00	0	0	0.00	0.00
1.1	842	0	179.12	84	218	28.07	0.00
ValidateToken
2.0	0	0	0.00	0	0	0.00	0.00
2.1	958	0	94.03	79	116	31.93	0.00
UserCRUD
3.0	0	0	0.00	0	0	0.00	0.00
3.1	0	0	0.00	0	0	0.00	0.00
3.2	1675	0	53.45	45	73	55.83	0.00
3.3	3	0	26.67	25	30	0.10	0.00
ProjectCRUD
4.0	0	0	0.00	0	0	0.00	0.00
4.1	0	0	0.00	0	0	0.00	0.00
4.2	1200	0	49.71	42	67	40.00	0.00
4.3	2	0	39.50	37	42	0.07	0.00
Aggregated	8666	0	68.94	25	218	288.87	0.00

Scenario Metrics

Transaction	# Users	# Times Run	Average (ms)	Min (ms)	Max (ms)	Scenarios/s	Iterations
ReadHeavy	7	1327	158.23	145	195	44.23	189.57
TokenLifecycle	5	837	179.37	166	218	27.90	167.40
ValidateToken	3	955	94.05	82	116	31.83	318.33
UserCRUD	3	1672	53.46	45	73	55.73	557.33
ProjectCRUD	2	1198	49.71	42	67	39.93	599.00
Aggregated	20	5989	100.00	42	218	199.63	1831.64

View full report

[github-actions]

Bencher Report

Branch	claude/storage-security-raft-review-e0n6vi
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
Command_Serde/apply/remove	📈 view plot 🚷 view threshold	118,310.00 ns (-58.16%) Baseline: 282,750.36 ns	1,686,273.19 ns (7.02%)
Command_Serde/apply/set	📈 view plot 🚷 view threshold	134,440.00 ns (-42.40%) Baseline: 233,398.47 ns	983,564.80 ns (13.67%)
Command_Serde/pack/delete	📈 view plot 🚷 view threshold	123.33 ns (+2.55%) Baseline: 120.27 ns	141.84 ns (86.95%)
Command_Serde/pack/delete_index	📈 view plot 🚷 view threshold	111.53 ns (+1.67%) Baseline: 109.70 ns	129.32 ns (86.24%)
Command_Serde/pack/set	📈 view plot 🚷 view threshold	211.77 ns (+9.83%) Baseline: 192.82 ns	230.80 ns (91.75%)
Command_Serde/pack/set_index	📈 view plot 🚷 view threshold	111.71 ns (+2.01%) Baseline: 109.51 ns	128.70 ns (86.80%)
Command_Serde/unpack/delete	📈 view plot 🚷 view threshold	205.55 ns (+6.57%) Baseline: 192.87 ns	233.96 ns (87.86%)
Command_Serde/unpack/delete_index	📈 view plot 🚷 view threshold	158.88 ns (-1.03%) Baseline: 160.54 ns	195.75 ns (81.16%)
Command_Serde/unpack/set	📈 view plot 🚷 view threshold	288.01 ns (+10.30%) Baseline: 261.10 ns	322.80 ns (89.22%)
Command_Serde/unpack/set_index	📈 view plot 🚷 view threshold	156.42 ns (-1.85%) Baseline: 159.37 ns	193.88 ns (80.68%)
Payload_encryption/pack/remove_cmd	📈 view plot 🚷 view threshold	125.79 ns (+8.02%) Baseline: 116.45 ns	138.79 ns (90.63%)
Payload_encryption/pack/set_cmd	📈 view plot 🚷 view threshold	218.89 ns (+6.63%) Baseline: 205.28 ns	271.75 ns (80.55%)
Payload_encryption/unpack/remove_cmd	📈 view plot 🚷 view threshold	215.04 ns (+5.00%) Baseline: 204.80 ns	247.03 ns (87.05%)
Payload_encryption/unpack/set_cmd	📈 view plot 🚷 view threshold	303.27 ns (+10.83%) Baseline: 273.63 ns	338.80 ns (89.51%)
Raft_1Node_Latency/prefix/1node	📈 view plot 🚷 view threshold	2,513,600.00 ns (-10.59%) Baseline: 2,811,277.66 ns	5,725,442.99 ns (43.90%)
Raft_1Node_Latency/read/1node	📈 view plot 🚷 view threshold	42,920.00 ns (+442.55%) Baseline: 7,910.76 ns	45,071.94 ns (95.23%)
Raft_1Node_Latency/remove/1node	📈 view plot 🚷 view threshold	349,420.00 ns (-36.05%) Baseline: 546,409.69 ns	2,294,915.48 ns (15.23%)
Raft_1Node_Latency/write/1node	📈 view plot 🚷 view threshold	370,060.00 ns (-34.64%) Baseline: 566,146.09 ns	2,127,959.36 ns (17.39%)
build_snapshot/default	📈 view plot 🚷 view threshold	114,010.00 ns (+8.54%) Baseline: 105,036.77 ns	161,656.81 ns (70.53%)
fernet token/project	📈 view plot 🚷 view threshold	1,426.20 ns (+2.96%) Baseline: 1,385.20 ns	1,623.34 ns (87.86%)
get_data_keyspace	📈 view plot 🚷 view threshold	0.31 ns (-1.20%) Baseline: 0.32 ns	0.37 ns (84.48%)
get_db	📈 view plot 🚷 view threshold	0.31 ns (-1.06%) Baseline: 0.32 ns	0.37 ns (84.71%)
get_fernet_token_timestamp/project	📈 view plot 🚷 view threshold	143.64 ns (-0.78%) Baseline: 144.77 ns	180.36 ns (79.64%)
get_keyspace	📈 view plot 🚷 view threshold	4.36 ns (-7.50%) Baseline: 4.71 ns	8.75 ns (49.80%)

🐰 View full continuous benchmarking report in Bencher