From 00399ef88d5b8dad161deace1096b1ce1b505b8d Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 5 Jul 2026 22:43:12 -0600 Subject: [PATCH 1/2] docs(analytics): document transaction queue depth metrics Add write-transaction-queue-depth and read-transaction-queue-depth to the Resource Usage metrics reference, with a section explaining the depth/maxDepth fields and alerting guidance (alert on the per-period maxDepth peak). Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/analytics/overview.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6..39fad3b0 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -175,10 +175,40 @@ Harper automatically tracks the following metrics for all services. Applications | ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | | `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | | `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | | `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | | `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | + +#### Transaction Queue Depth Metrics + +`write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are +in flight against the storage engine, giving operators a leading indicator before a write-heavy +workload hits the `Outstanding write transactions have too long of queue, please try again later` +(HTTP 503) rejection. + +| Field | Unit | Description | +| ---------- | ----- | -------------------------------------------------------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark observed over the sampling period | + +- **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet + durably committed — the backlog that produces the overload error when it drains too slowly. +- **`read-transaction-queue-depth`** counts open read (snapshot) transactions. Persistently high read + depth indicates long-lived read snapshots, which can hold back compaction. + +Both are gauges sampled per worker thread and summed across threads in the aggregate table (the raw +per-thread entries in `hdb_raw_analytics` retain each thread's own depth). Note that the aggregate +`maxDepth` is the sum of each thread's peak, which can read higher than any true simultaneous global +peak since per-thread spikes need not coincide — treat it as an upper bound, not an exact concurrent +queue length. Because the write queue can fill and drain within a single sampling period, always +alert on `maxDepth` (the per-period peak) rather than `depth` alone — an instantaneous sample will +routinely read low even while short spikes are occurring. A healthy system keeps `write-transaction-queue-depth.maxDepth` +near zero; a sustained non-zero peak that trends upward is the signal to shed or throttle write load +before commits start timing out. Tune the concrete alert threshold against a baseline for your +workload, since absolute depth scales with worker-thread count and per-transaction size. #### `resource-usage` Metric From 9a8f49f3bd95a3917921651a6df4accbb8d12f22 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 10:29:20 -0600 Subject: [PATCH 2/2] fix(analytics): satisfy prettier table formatting --- reference/analytics/overview.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 39fad3b0..3dc19060 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,16 +171,16 @@ Harper automatically tracks the following metrics for all services. Applications ### Resource Usage Metrics -| `metric` | Key attributes | Other | Unit | Description | -| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | -| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | -| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | -| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | -| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | -| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | -| `table-size` | `size` | `database`, `table` | bytes | Table file size | -| `utilization` | | | % | Percentage of time the worker thread was processing requests | -| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `metric` | Key attributes | Other | Unit | Description | +| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | ----------------------------------------------------------------------------------------------------- | +| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | +| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | +| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | +| `table-size` | `size` | `database`, `table` | bytes | Table file size | +| `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | #### Transaction Queue Depth Metrics @@ -189,10 +189,10 @@ in flight against the storage engine, giving operators a leading indicator befor workload hits the `Outstanding write transactions have too long of queue, please try again later` (HTTP 503) rejection. -| Field | Unit | Description | -| ---------- | ----- | -------------------------------------------------------------------------------------------------- | -| `depth` | count | Instantaneous depth sampled at emit time | -| `maxDepth` | count | High-water mark observed over the sampling period | +| Field | Unit | Description | +| ---------- | ----- | ------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark observed over the sampling period | - **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet durably committed — the backlog that produces the overload error when it drains too slowly.