diff --git a/docs/admin/code-hosts/aws-codecommit.mdx b/docs/admin/code-hosts/aws-codecommit.mdx index 5f1666ebc..7b8736801 100644 --- a/docs/admin/code-hosts/aws-codecommit.mdx +++ b/docs/admin/code-hosts/aws-codecommit.mdx @@ -37,7 +37,7 @@ AWS CodeCommit connections support the following configuration options, which ar {/* SCHEMA_SYNC_START: admin/code_hosts/aws_codecommit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // REQUIRED: diff --git a/docs/admin/code-hosts/azuredevops.mdx b/docs/admin/code-hosts/azuredevops.mdx index 7695f459c..0bfa23955 100644 --- a/docs/admin/code-hosts/azuredevops.mdx +++ b/docs/admin/code-hosts/azuredevops.mdx @@ -69,7 +69,7 @@ Azure DevOps connections support the following configuration options, which are {/* SCHEMA_SYNC_START: admin/code_hosts/azuredevops.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json // Authentication alternatives: token OR windowsPassword diff --git a/docs/admin/code-hosts/bitbucket-cloud.mdx b/docs/admin/code-hosts/bitbucket-cloud.mdx index 51070727f..0bc8b1dcd 100644 --- a/docs/admin/code-hosts/bitbucket-cloud.mdx +++ b/docs/admin/code-hosts/bitbucket-cloud.mdx @@ -116,7 +116,7 @@ Bitbucket Cloud connections support the following configuration options, which a {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_cloud.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // The workspace access token to use when authenticating with Bitbucket Cloud. diff --git a/docs/admin/code-hosts/bitbucket-server.mdx b/docs/admin/code-hosts/bitbucket-server.mdx index 6775b7dbb..76369454e 100644 --- a/docs/admin/code-hosts/bitbucket-server.mdx +++ b/docs/admin/code-hosts/bitbucket-server.mdx @@ -202,7 +202,7 @@ Bitbucket Server / Bitbucket Data Center connections support the following confi {/* SCHEMA_SYNC_START: admin/code_hosts/bitbucket_server.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json // Authentication alternatives: token OR password diff --git a/docs/admin/code-hosts/gerrit.mdx b/docs/admin/code-hosts/gerrit.mdx index f20e399f3..be216129b 100644 --- a/docs/admin/code-hosts/gerrit.mdx +++ b/docs/admin/code-hosts/gerrit.mdx @@ -113,7 +113,7 @@ Gerrit connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/gerrit.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // If non-null, enforces Gerrit repository permissions. This requires that there is an item in the [site configuration json](https://sourcegraph.com/docs/admin/config/site_config#auth-providers) `auth.providers` field, of type "gerrit" with the same `url` field as specified in this `GerritConnection`. diff --git a/docs/admin/code-hosts/github.mdx b/docs/admin/code-hosts/github.mdx index fc0144103..8fb45319d 100644 --- a/docs/admin/code-hosts/github.mdx +++ b/docs/admin/code-hosts/github.mdx @@ -454,7 +454,7 @@ GitHub connections support the following configuration options, which are specif {/* SCHEMA_SYNC_START: admin/code_hosts/github.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json // Authentication alternatives: token OR gitHubAppDetails OR externalAccount OR useRandomExternalAccount diff --git a/docs/admin/code-hosts/gitlab.mdx b/docs/admin/code-hosts/gitlab.mdx index 518ecea02..b4b840a44 100644 --- a/docs/admin/code-hosts/gitlab.mdx +++ b/docs/admin/code-hosts/gitlab.mdx @@ -189,7 +189,7 @@ See [Internal rate limits](/admin/code-hosts/rate-limits#internal-rate-limits). {/* SCHEMA_SYNC_START: admin/code_hosts/gitlab.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // If non-null, enforces GitLab repository permissions. This requires that there be an item in the `auth.providers` field of type "gitlab" with the same `url` field as specified in this `GitLabConnection`. diff --git a/docs/admin/code-hosts/gitolite.mdx b/docs/admin/code-hosts/gitolite.mdx index f6837c8f2..405ab81e2 100644 --- a/docs/admin/code-hosts/gitolite.mdx +++ b/docs/admin/code-hosts/gitolite.mdx @@ -25,7 +25,7 @@ To connect Gitolite to Sourcegraph: {/* SCHEMA_SYNC_START: admin/code_hosts/gitolite.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // A list of repositories to never mirror from this Gitolite instance. Supports excluding by exact name ({"name": "foo"}). diff --git a/docs/admin/code-hosts/other.mdx b/docs/admin/code-hosts/other.mdx index f92d971c6..3495ca10a 100644 --- a/docs/admin/code-hosts/other.mdx +++ b/docs/admin/code-hosts/other.mdx @@ -68,7 +68,7 @@ Repositories must be listed individually: {/* SCHEMA_SYNC_START: admin/code_hosts/other_external_service.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // A list of repositories to never mirror by name after applying repositoryPathPattern. Supports excluding by exact name ({"name": "myrepo"}) or regular expression ({"pattern": ".*secret.*"}). diff --git a/docs/admin/code-hosts/phabricator.mdx b/docs/admin/code-hosts/phabricator.mdx index 3f1c489b8..7585c9612 100644 --- a/docs/admin/code-hosts/phabricator.mdx +++ b/docs/admin/code-hosts/phabricator.mdx @@ -76,7 +76,7 @@ The Sourcegraph instance's site admin must [update the `corsOrigin` site config {/* SCHEMA_SYNC_START: admin/code_hosts/phabricator.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // SSH cipher to use when cloning via SSH. Must be a valid choice from `ssh -Q cipher`. diff --git a/docs/admin/config/settings.mdx b/docs/admin/config/settings.mdx index 7fd8d1822..2f9ab6739 100644 --- a/docs/admin/config/settings.mdx +++ b/docs/admin/config/settings.mdx @@ -27,7 +27,7 @@ Settings options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/settings.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { diff --git a/docs/admin/config/site-config.mdx b/docs/admin/config/site-config.mdx index 6904c12de..7ccd9eac3 100644 --- a/docs/admin/config/site-config.mdx +++ b/docs/admin/config/site-config.mdx @@ -21,7 +21,7 @@ All site configuration options and their default values are shown below. {/* SCHEMA_SYNC_START: admin/config/site.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { diff --git a/docs/admin/repo/perforce.mdx b/docs/admin/repo/perforce.mdx index 5adb427f1..851f08a23 100644 --- a/docs/admin/repo/perforce.mdx +++ b/docs/admin/repo/perforce.mdx @@ -228,7 +228,7 @@ With this setting, Sourcegraph will ignore any rules with a host other than `*`, {/* SCHEMA_SYNC_START: admin/code_hosts/perforce.schema.json */} {/* WARNING: This section is auto-generated during releases. Do not edit manually. */} -{/* Last updated: 2026-08-05T21:41:22Z */} +{/* Last updated: 2026-08-11T22:50:09Z */} ```json { // If non-null, enforces Perforce depot permissions. diff --git a/docs/self-hosted/observability/alerts.mdx b/docs/self-hosted/observability/alerts.mdx index db6679c77..5fb55298c 100644 --- a/docs/self-hosted/observability/alerts.mdx +++ b/docs/self-hosted/observability/alerts.mdx @@ -3607,6 +3607,43 @@ Generated query for critical alert: `max((max(src_repoupdater_cleanup_failed_rep
+## worker: entitlements_usage_history_telemetry_queue_oldest_age + +

age of the oldest unexported entitlement usage history event

+ +**Descriptions** + +- warning worker: 1800s+ age of the oldest unexported entitlement usage history event for 30m0s +- critical worker: 7200s+ age of the oldest unexported entitlement usage history event for 2h0m0s + +**Next steps** + +- Check whether the entitlement usage history telemetry export job is failing or making no forward progress. +- Compare this panel with the exported events and export batch size panels above. +- Check worker logs for entitlements usage history telemetry exporter failures. +- More help interpreting this metric is available in the [dashboards reference](dashboards#worker-entitlements_usage_history_telemetry_queue_oldest_age). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_entitlements_usage_history_telemetry_queue_oldest_age", + "critical_worker_entitlements_usage_history_telemetry_queue_oldest_age" +] +``` + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Generated query for warning alert: `max(((max(src_entitlementstelemetry_usage_history_queue_oldest_age_seconds and on (instance, job) ((increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"\}[15m]) > 0) or on (instance, job) (increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total\{job=~"^worker.*"}[15m]) > 0)))) or on () vector(0)) > 1800)` + +Generated query for critical alert: `max(((max(src_entitlementstelemetry_usage_history_queue_oldest_age_seconds and on (instance, job) ((increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"\}[15m]) > 0) or on (instance, job) (increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total\{job=~"^worker.*"}[15m]) > 0)))) or on () vector(0)) > 7200)` + +
+ +
+ ## worker: src_repoupdater_external_services_total

the total number of external services

@@ -6524,6 +6561,42 @@ Generated query for warning alert: `max((max by (sg_instance) (go_gc_duration_se
+## codeintel-uploads: codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age + +

approximate age of the oldest SCIP document dereference log record, based on the lowest-ID record

+ +**Descriptions** + +- warning codeintel-uploads: 604800s+ approximate age of the oldest SCIP document dereference log record, based on the lowest-ID record for 15m0s + +**Next steps** + +- Check the SCIP documents janitor error rate and duration panels for failures or slow cleanup queries. +- Check sum(rate(src_codeintel_uploads_janitor_scip_documents_records_scanned_total[5m])). A sustained non-zero rate means the janitor is processing eligible dereference logs. +- Check sum(rate(src_codeintel_uploads_janitor_scip_documents_records_altered_total[5m])) for the document deletion rate. This may legitimately be lower than the scanned rate because referenced documents are retained. +- Check codeintel-db CPU, I/O, locks, and table growth for codeintel_scip_documents_dereference_logs. +- If cleanup is healthy but cannot keep up with the volume of generated logs, increase CODEINTEL_UPLOADS_UNREFERENCED_DOCUMENT_MAX_BATCHES. +- If increasing the batch budget is insufficient and codeintel-db has capacity, increase CODEINTEL_UPLOADS_UNREFERENCED_DOCUMENT_MAX_CONCURRENT_BATCHES. +- Learn more about the related dashboard panel in the [dashboards reference](dashboards#codeintel-uploads-codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age). +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_codeintel-uploads_codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age" +] +``` + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Generated query for warning alert: `max(((time() - max(src_codeintel_uploads_janitor_scip_documents_oldest_dereference_log_timestamp_seconds)) and max(src_codeintel_uploads_janitor_scip_documents_oldest_dereference_log_timestamp_seconds) > 0) > 604800)` + +
+ +
+ ## telemetry: telemetry_gateway_exporter_queue_growth

rate of growth of events export queue over 30m

diff --git a/docs/self-hosted/observability/dashboards.mdx b/docs/self-hosted/observability/dashboards.mdx index 6dc83214b..7bdeb2b97 100644 --- a/docs/self-hosted/observability/dashboards.mdx +++ b/docs/self-hosted/observability/dashboards.mdx @@ -15428,6 +15428,157 @@ sum by (reason) (src_repo_statesyncer_repos_deleted{is_primary="false"})
+### Worker: Entitlements usage history telemetry: events export and queue metrics + +#### worker: entitlements_usage_history_telemetry_exported_events + +

Entitlement usage history events exported per hour

+ +The number of entitlement usage history events exported as telemetry over the last hour. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101200` on your Sourcegraph instance. + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Query: + +``` +sum(increase(src_entitlementstelemetry_usage_history_exported_events[1h])) +``` +
+ +
+ +#### worker: entitlements_usage_history_telemetry_batch_size + +

Number of entitlement usage history events exported per batch over 30m

+ +The number of entitlement usage history events exported in each batch. The largest bucket is the maximum number of events exported per batch. +If the distribution trends to the maximum bucket, export throughput is at or approaching saturation. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101201` on your Sourcegraph instance. + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Query: + +``` +sum by (le) (rate(src_entitlementstelemetry_usage_history_batch_size_bucket[30m])) +``` +
+ +
+ +#### worker: entitlements_usage_history_telemetry_pruned_events + +

Entitlement usage history events pruned per hour

+ +The number of already-exported entitlement usage history events pruned over the last hour. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101210` on your Sourcegraph instance. + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Query: + +``` +sum(increase(src_entitlementstelemetry_usage_history_events_pruned[1h])) +``` +
+ +
+ +#### worker: entitlements_usage_history_telemetry_queue_size + +

Entitlement usage history events pending telemetry export

+ +The number of entitlement usage history events waiting to be exported as telemetry. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101211` on your Sourcegraph instance. + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Query: + +``` +( + sum( + src_entitlementstelemetry_usage_history_queue_size + and on(instance, job) + ( + ( + increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"}[15m]) > 0 + ) + or on(instance, job) + ( + increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total{job=~"^worker.*"}[15m]) > 0 + ) + ) + ) + ) or on() vector(0) +``` +
+ +
+ +#### worker: entitlements_usage_history_telemetry_queue_oldest_age + +

Age of the oldest unexported entitlement usage history event

+ +The age in seconds of the oldest entitlement usage history event waiting to be exported as telemetry. + +Refer to the [alerts reference](alerts#worker-entitlements_usage_history_telemetry_queue_oldest_age) for 2 alerts related to this panel. + +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101220` on your Sourcegraph instance. + +*Managed by the Sourcegraph Services team.* + +
+Technical details + +Query: + +``` +( + max( + src_entitlementstelemetry_usage_history_queue_oldest_age_seconds + and on(instance, job) + ( + ( + increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"}[15m]) > 0 + ) + or on(instance, job) + ( + increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total{job=~"^worker.*"}[15m]) > 0 + ) + ) + ) + ) or on() vector(0) +``` +
+ +
+ ### Worker: External services #### worker: src_repoupdater_external_services_total @@ -15436,7 +15587,7 @@ sum by (reason) (src_repo_statesyncer_repos_deleted{is_primary="false"}) Refer to the [alerts reference](alerts#worker-src_repoupdater_external_services_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101200` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101300` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15458,7 +15609,7 @@ max(src_repoupdater_external_services_total) Refer to the [alerts reference](alerts#worker-repoupdater_queued_sync_jobs_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101210` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101310` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15480,7 +15631,7 @@ max(src_repoupdater_queued_sync_jobs_total) Refer to the [alerts reference](alerts#worker-repoupdater_completed_sync_jobs_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101211` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101311` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15502,7 +15653,7 @@ max(src_repoupdater_completed_sync_jobs_total) Refer to the [alerts reference](alerts#worker-repoupdater_errored_sync_jobs_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101212` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101312` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15524,7 +15675,7 @@ max(src_repoupdater_errored_sync_jobs_percentage) Refer to the [alerts reference](alerts#worker-github_graphql_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101220` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101320` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15546,7 +15697,7 @@ max by (name) (src_github_rate_limit_remaining_v2{resource="graphql"}) Refer to the [alerts reference](alerts#worker-github_rest_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101221` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101321` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15568,7 +15719,7 @@ max by (name) (src_github_rate_limit_remaining_v2{resource="rest"}) Refer to the [alerts reference](alerts#worker-github_search_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101222` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101322` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15592,7 +15743,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101230` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101330` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15616,7 +15767,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101231` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101331` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15640,7 +15791,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101232` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101332` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15662,7 +15813,7 @@ max by(name) (rate(src_github_rate_limit_wait_duration_seconds{resource="search" Refer to the [alerts reference](alerts#worker-gitlab_rest_rate_limit_remaining) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101240` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101340` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15686,7 +15837,7 @@ Indicates how long we`re waiting on the rate limit once it has been exceeded This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101241` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101341` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15710,7 +15861,7 @@ Indicates how long we`re waiting on our internal rate limiter when communicating This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101250` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101350` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15734,7 +15885,7 @@ The rate at which we fail our internal rate limiter. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101251` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101351` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15760,7 +15911,7 @@ Indicates the total number of user permissions sync completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101400` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15784,7 +15935,7 @@ Indicates the number of users permissions syncs completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101401` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15808,7 +15959,7 @@ Indicates the number of permissions syncs done for the first time for the user. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101302` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101402` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15832,7 +15983,7 @@ Indicates the total number of repo permissions sync completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101310` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101410` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15856,7 +16007,7 @@ Indicates the number of repos permissions syncs completed. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101311` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101411` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15880,7 +16031,7 @@ Indicates the number of permissions syncs done for the first time for the repo. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101312` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101412` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15904,7 +16055,7 @@ Indicates the max delay between two consecutive permissions sync for a user duri This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101320` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101420` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15928,7 +16079,7 @@ Indicates the max delay between two consecutive permissions sync for a repo duri This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101321` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101421` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15952,7 +16103,7 @@ Indicates the max delay between user creation and their permissions sync This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101330` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101430` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -15976,7 +16127,7 @@ Indicates the max delay between repo creation and their permissions sync This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101331` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101431` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16000,7 +16151,7 @@ Indicates the number permissions found during users/repos permissions sync. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101340` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101440` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16024,7 +16175,7 @@ Indicates the average number permissions found during permissions sync per user/ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101341` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101441` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16046,7 +16197,7 @@ avg by (type) (src_repo_perms_syncer_perms_found) Refer to the [alerts reference](alerts#worker-perms_syncer_outdated_perms) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101350` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101450` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16068,7 +16219,7 @@ max by (type) (src_repo_perms_syncer_outdated_perms) Refer to the [alerts reference](alerts#worker-perms_syncer_sync_duration) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101360` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101460` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16094,7 +16245,7 @@ Permissions sync errors are often transient and rarely actionable. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101370` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101470` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16119,7 +16270,7 @@ More about repository permissions synchronization [here](https://sourcegraph.com This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101371` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101471` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16143,7 +16294,7 @@ sum by (type) (increase(src_repo_perms_syncer_schedule_total[1m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101400` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101500` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16165,7 +16316,7 @@ sum(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101401` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101501` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16187,7 +16338,7 @@ sum by (le)(rate(src_gitserver_client_duration_seconds_bucket{job=~"^worker.*"} This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101402` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101502` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16209,7 +16360,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101403` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101503` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16231,7 +16382,7 @@ sum(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5m])) / (sum(in This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101410` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101510` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16253,7 +16404,7 @@ sum by (op,scope)(increase(src_gitserver_client_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101411` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101511` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16275,7 +16426,7 @@ histogram_quantile(0.99, sum by (le,op,scope)(rate(src_gitserver_client_duratio This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101412` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101512` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16297,7 +16448,7 @@ sum by (op,scope)(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101413` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101513` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16321,7 +16472,7 @@ sum by (op,scope)(increase(src_gitserver_client_errors_total{job=~"^worker.*"}[5 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101500` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101600` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16343,7 +16494,7 @@ sum(increase(src_gitserver_repositoryservice_client_total{job=~"^worker.*"}[5m]) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101501` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101601` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16365,7 +16516,7 @@ sum by (le)(rate(src_gitserver_repositoryservice_client_duration_seconds_bucket This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101502` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101602` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16387,7 +16538,7 @@ sum(increase(src_gitserver_repositoryservice_client_errors_total{job=~"^worker.* This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101503` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101603` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16409,7 +16560,7 @@ sum(increase(src_gitserver_repositoryservice_client_errors_total{job=~"^worker.* This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101510` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101610` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16431,7 +16582,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_total{job=~"^w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101511` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101611` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16453,7 +16604,7 @@ histogram_quantile(0.99, sum by (le,op,scope)(rate(src_gitserver_repositoryserv This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101512` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101612` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16475,7 +16626,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_errors_total{j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101513` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101613` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -16499,7 +16650,7 @@ sum by (op,scope)(increase(src_gitserver_repositoryservice_client_errors_total{j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101600` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101700` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16521,7 +16672,7 @@ sum(increase(src_batches_dbstore_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101601` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101701` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16543,7 +16694,7 @@ sum by (le)(rate(src_batches_dbstore_duration_seconds_bucket{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101602` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101702` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16565,7 +16716,7 @@ sum(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101603` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101703` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16587,7 +16738,7 @@ sum(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) / (sum(inc This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101610` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101710` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16609,7 +16760,7 @@ sum by (op)(increase(src_batches_dbstore_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101611` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101711` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16631,7 +16782,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_batches_dbstore_duration_secon This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101612` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101712` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16653,7 +16804,7 @@ sum by (op)(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101613` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101713` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16677,7 +16828,7 @@ sum by (op)(increase(src_batches_dbstore_errors_total{job=~"^worker.*"}[5m])) / This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101700` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101800` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16699,7 +16850,7 @@ sum(increase(src_batches_service_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101701` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101801` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16721,7 +16872,7 @@ sum by (le)(rate(src_batches_service_duration_seconds_bucket{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101702` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101802` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16743,7 +16894,7 @@ sum(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101703` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101803` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16765,7 +16916,7 @@ sum(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) / (sum(inc This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101710` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101810` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16787,7 +16938,7 @@ sum by (op)(increase(src_batches_service_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101711` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101811` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16809,7 +16960,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_batches_service_duration_secon This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101712` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101812` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16831,7 +16982,7 @@ sum by (op)(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101713` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101813` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16855,7 +17006,7 @@ sum by (op)(increase(src_batches_service_errors_total{job=~"^worker.*"}[5m])) / This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101800` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101900` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16877,7 +17028,7 @@ sum(src_query_runner_worker_processor_handlers{job=~"^worker.*"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101810` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101910` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16899,7 +17050,7 @@ sum(increase(src_query_runner_worker_processor_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101811` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101911` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16921,7 +17072,7 @@ sum by (le)(rate(src_query_runner_worker_processor_duration_seconds_bucket{job= This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101812` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101912` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16943,7 +17094,7 @@ sum(increase(src_query_runner_worker_processor_errors_total{job=~"^worker.*"}[5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101813` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101913` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16967,7 +17118,7 @@ sum(increase(src_query_runner_worker_processor_errors_total{job=~"^worker.*"}[5m This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101900` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102000` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -16989,7 +17140,7 @@ sum(increase(src_workerutil_dbworker_store_total{domain='insights_query_runner_j This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101901` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102001` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17011,7 +17162,7 @@ sum by (le)(rate(src_workerutil_dbworker_store_duration_seconds_bucket{domain=' This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101902` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102002` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17033,7 +17184,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='insights_query_r This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101903` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102003` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17055,7 +17206,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='insights_query_r This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101910` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102010` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17077,7 +17228,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_total{domain='insights_query_ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101911` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102011` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17099,7 +17250,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_workerutil_dbworker_store_dura This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101912` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102012` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17121,7 +17272,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='insights This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=101913` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102013` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17148,7 +17299,7 @@ A value of 0 indicates the routine isn`t running currently, it awaits it`s next This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102000` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102100` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17173,7 +17324,7 @@ A low or zero value could indicate that a routine is stalled or encountering err This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102001` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102101` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17198,7 +17349,7 @@ A sustained high error rate may indicate a problem with the routine`s configurat Refer to the [alerts reference](alerts#worker-goroutine_error_rate) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102010` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102110` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -17223,7 +17374,7 @@ A value above 5% indicates that a significant portion of routine executions are Refer to the [alerts reference](alerts#worker-goroutine_error_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102011` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102111` on your Sourcegraph instance. *Managed by the Sourcegraph Services team.* @@ -17248,7 +17399,7 @@ Longer durations might indicate increased load or processing time. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102020` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102120` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17273,7 +17424,7 @@ This represents how long a complete loop iteration takes before sleeping for the This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102021` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102121` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17298,7 +17449,7 @@ Higher values indicate that tenant processing is taking longer and may affect ov This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102030` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102130` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17323,7 +17474,7 @@ Consistently high values might indicate problematic tenants or inefficient proce This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102031` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102131` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17348,7 +17499,7 @@ Unexpected changes can indicate tenant configuration issues or scaling events. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102040` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102140` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17373,7 +17524,7 @@ A healthy routine should maintain a consistent processing rate. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102041` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102141` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17398,7 +17549,7 @@ Consistent errors indicate problems with specific tenants. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102050` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102150` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17423,7 +17574,7 @@ Values above 5% indicate significant tenant processing problems. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102051` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102151` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17447,7 +17598,7 @@ Query: This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102100` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102200` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17469,7 +17620,7 @@ sum by (app_name, db_name) (src_pgsql_conns_max_open{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102101` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102201` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17491,7 +17642,7 @@ sum by (app_name, db_name) (src_pgsql_conns_open{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102110` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102210` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17513,7 +17664,7 @@ sum by (app_name, db_name) (src_pgsql_conns_in_use{app_name="worker"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102111` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102211` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17535,7 +17686,7 @@ sum by (app_name, db_name) (src_pgsql_conns_idle{app_name="worker"}) Refer to the [alerts reference](alerts#worker-mean_blocked_seconds_per_conn_request) for 2 alerts related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102120` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102220` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17557,7 +17708,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102130` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102230` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17579,7 +17730,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle{app_name="w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102131` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102231` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17601,7 +17752,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_lifetime{app_nam This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102132` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102232` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -17625,7 +17776,7 @@ sum by (app_name, db_name) (increase(src_pgsql_conns_closed_max_idle_time{app_na Refer to the [alerts reference](alerts#worker-cpu_usage_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102200` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102300` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17649,7 +17800,7 @@ An estimate for the active memory in use, which includes anonymous memory, file This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102201` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102301` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17673,7 +17824,7 @@ An estimate for the active memory in use in bytes, which includes anonymous memo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102202` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102302` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17697,7 +17848,7 @@ The total anonymous memory in use by the application, which includes Go stack an Refer to the [alerts reference](alerts#worker-memory_rss) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102210` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102310` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17721,7 +17872,7 @@ This metric shows the total active file-backed memory currently in use by the ap This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102211` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102311` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17745,7 +17896,7 @@ The kernel usage metric shows the amount of memory used by the kernel on behalf This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102212` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102312` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17779,7 +17930,7 @@ value change independent of deployment events (such as an upgrade), it could ind This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102400` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17801,7 +17952,7 @@ count by(name) ((time() - container_last_seen{name=~"^worker.*"}) > 60) Refer to the [alerts reference](alerts#worker-container_cpu_usage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102401` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17823,7 +17974,7 @@ cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"} Refer to the [alerts reference](alerts#worker-container_memory_usage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102302` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102402` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17848,7 +17999,7 @@ When extremely high, this can indicate a resource usage problem, or can cause pr This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102303` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102403` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -17872,7 +18023,7 @@ sum by(name) (rate(container_fs_reads_total{name=~"^worker.*"}[1h]) + rate(conta Refer to the [alerts reference](alerts#worker-provisioning_container_cpu_usage_long_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102400` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102500` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17894,7 +18045,7 @@ quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^wo Refer to the [alerts reference](alerts#worker-provisioning_container_memory_usage_long_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102401` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102501` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17916,7 +18067,7 @@ max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*" Refer to the [alerts reference](alerts#worker-provisioning_container_cpu_usage_short_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102410` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102510` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17938,7 +18089,7 @@ max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[5 Refer to the [alerts reference](alerts#worker-provisioning_container_memory_usage_short_term) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102411` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102511` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17963,7 +18114,7 @@ When it occurs frequently, it is an indicator of underprovisioning. Refer to the [alerts reference](alerts#worker-container_oomkill_events_total) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102412` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102512` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -17989,7 +18140,7 @@ A high value here indicates a possible goroutine leak. Refer to the [alerts reference](alerts#worker-go_goroutines) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102500` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102600` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18011,7 +18162,7 @@ max by(instance) (go_goroutines{job=~".*worker"}) Refer to the [alerts reference](alerts#worker-go_gc_duration_seconds) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102501` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102601` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18035,7 +18186,7 @@ max by(instance) (go_gc_duration_seconds{job=~".*worker"}) Refer to the [alerts reference](alerts#worker-pods_available_percentage) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102600` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102700` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18059,7 +18210,7 @@ sum by(app) (up{app=~".*worker"}) / count by (app) (up{app=~".*worker"}) * 100 This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102700` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102800` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18081,7 +18232,7 @@ sum(increase(src_workerutil_dbworker_store_total{domain='own_background_worker_s This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102701` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102801` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18103,7 +18254,7 @@ sum by (le)(rate(src_workerutil_dbworker_store_duration_seconds_bucket{domain=' This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102702` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102802` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18125,7 +18276,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='own_background_w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102703` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102803` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18147,7 +18298,7 @@ sum(increase(src_workerutil_dbworker_store_errors_total{domain='own_background_w This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102710` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102810` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18169,7 +18320,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_total{domain='own_background_ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102711` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102811` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18191,7 +18342,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_workerutil_dbworker_store_dura This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102712` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102812` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18213,7 +18364,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='own_back This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102713` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102813` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18237,7 +18388,7 @@ sum by (op)(increase(src_workerutil_dbworker_store_errors_total{domain='own_back This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102800` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102900` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18259,7 +18410,7 @@ sum(src_own_background_worker_processor_handlers{job=~"^worker.*"}) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102810` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102910` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18281,7 +18432,7 @@ sum(increase(src_own_background_worker_processor_total{job=~"^worker.*"}[5m])) This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102811` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102911` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18303,7 +18454,7 @@ sum by (le)(rate(src_own_background_worker_processor_duration_seconds_bucket{jo This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102812` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102912` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18325,7 +18476,7 @@ sum(increase(src_own_background_worker_processor_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102813` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102913` on your Sourcegraph instance. *Managed by the Sourcegraph Code Plane team.* @@ -18349,7 +18500,7 @@ sum(increase(src_own_background_worker_processor_errors_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102900` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103000` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18371,7 +18522,7 @@ sum by (op)(increase(src_own_background_index_scheduler_total{job=~"^worker.*"}[ This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102901` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103001` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18393,7 +18544,7 @@ histogram_quantile(0.99, sum by (le,op)(rate(src_own_background_index_scheduler This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102902` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103002` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18415,7 +18566,7 @@ sum by (op)(increase(src_own_background_index_scheduler_errors_total{job=~"^work This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=102903` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103003` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -18441,7 +18592,7 @@ The duration since the configuration client used by the "worker" service last su This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103000` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103100` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -18463,7 +18614,7 @@ src_conf_client_time_since_last_successful_update_seconds{job=~`^worker.*`,insta Refer to the [alerts reference](alerts#worker-worker_site_configuration_duration_since_last_successful_update_by_instance) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103001` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/worker/worker?viewPanel=103101` on your Sourcegraph instance. *Managed by the Sourcegraph Platform team.* @@ -26245,7 +26396,7 @@ max by (name, queue) (src_executors_managed_instance_groups_autoscaler_assigned_

Codeintel executor queue has a backlog that is not being processed

-Fires when the codeintel executor queue has a non-empty backlog (`src_workerutil_queue_depth{domain="executors_codeintel"} > 0`) while zero jobs have reached a terminal state over the last 30 minutes (`src_workerutil_dbworker_store_total{domain="codeintel_index"}`). With a backlog present, zero completions over 30m is already abnormal — the only benign zero-completion case (an empty queue) is excluded by the backlog guard. +Fires when the codeintel executor queue has a non-empty backlog (src_workerutil_queue_depth{domain="executors_codeintel"\} > 0) while zero jobs have reached a terminal state over the last 30 minutes (src_workerutil_dbworker_store_total\{domain="codeintel_index"}). With a backlog present, zero completions over 30m is already abnormal — the only benign zero-completion case (an empty queue) is excluded by the backlog guard. This is the true "executors not draining" signal: it distinguishes stuck/broken executors from executors that are simply under-provisioned. An under-provisioned queue (backlog present but jobs still completing) has a non-zero terminal-transition rate and will NOT trip this alert — that case is a capacity-review signal, not a stall. Both signals come from the worker/frontend (always scraped), not from the executor VMs, so this alert works on deployments where executors run outside the cluster (e.g. Sourcegraph Cloud). @@ -26271,7 +26422,7 @@ sum(rate(src_workerutil_dbworker_store_total{domain="codeintel_index",op=~"MarkC

Batches executor queue has a backlog that is not being processed

-Fires when the batches executor queue has a non-empty backlog (`src_workerutil_queue_depth{domain="executors_batches"} > 0`) while zero jobs have reached a terminal state over the last 30 minutes (`src_workerutil_dbworker_store_total{domain="batch_spec_workspace_execution_worker_store"}`). With a backlog present, zero completions over 30m is already abnormal — the only benign zero-completion case (an empty queue) is excluded by the backlog guard. +Fires when the batches executor queue has a non-empty backlog (src_workerutil_queue_depth{domain="executors_batches"\} > 0) while zero jobs have reached a terminal state over the last 30 minutes (src_workerutil_dbworker_store_total\{domain="batch_spec_workspace_execution_worker_store"}). With a backlog present, zero completions over 30m is already abnormal — the only benign zero-completion case (an empty queue) is excluded by the backlog guard. This is the true "executors not draining" signal: it distinguishes stuck/broken executors from executors that are simply under-provisioned. An under-provisioned queue (backlog present but jobs still completing) has a non-zero terminal-transition rate and will NOT trip this alert — that case is a capacity-review signal, not a stall. Both signals come from the worker/frontend (always scraped), not from the executor VMs, so this alert works on deployments where executors run outside the cluster (e.g. Sourcegraph Cloud). @@ -32779,11 +32930,35 @@ sum by (op)(increase(src_codeintel_uploads_janitor_audit_logs_errors_total{job=~

Records scanned every 5m

-The number of candidate records considered for cleanup. +The number of candidate records considered for cleanup. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads?viewPanel=101300` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum(increase(src_codeintel_uploads_janitor_scip_documents_records_scanned_total{job=~"^${source:regex}.*"}[5m])) +``` +
+ +
+ +#### codeintel-uploads: codeintel_uploads_janitor_scip_documents_records_altered_total + +

Records altered every 5m

+ +The number of candidate records altered as part of cleanup. This panel has no related alerts. -To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads?viewPanel=101300` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads?viewPanel=101301` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -32793,21 +32968,19 @@ To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads Query: ``` -sum(increase(src_codeintel_uploads_janitor_scip_documents_records_scanned_total{job=~"^${source:regex}.*"}[5m])) +sum(increase(src_codeintel_uploads_janitor_scip_documents_records_altered_total{job=~"^${source:regex}.*"}[5m])) ```
-#### codeintel-uploads: codeintel_uploads_janitor_scip_documents_records_altered_total - -

Records altered every 5m

+#### codeintel-uploads: codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age -The number of candidate records altered as part of cleanup. +

Approximate age of the oldest SCIP document dereference log record, based on the lowest-ID record

-This panel has no related alerts. +Refer to the [alerts reference](alerts#codeintel-uploads-codeintel_uploads_janitor_scip_documents_oldest_dereference_log_age) for 1 alert related to this panel. -To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads?viewPanel=101301` on your Sourcegraph instance. +To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads?viewPanel=101302` on your Sourcegraph instance. *Managed by the Sourcegraph Code Understanding team.* @@ -32817,7 +32990,7 @@ To see this panel, visit `/-/debug/grafana/d/codeintel-uploads/codeintel-uploads Query: ``` -sum(increase(src_codeintel_uploads_janitor_scip_documents_records_altered_total{job=~"^${source:regex}.*"}[5m])) +(time() - max(src_codeintel_uploads_janitor_scip_documents_oldest_dereference_log_timestamp_seconds{job=~"^${source:regex}.*"})) and max(src_codeintel_uploads_janitor_scip_documents_oldest_dereference_log_timestamp_seconds{job=~"^${source:regex}.*"}) > 0 ``` @@ -35075,163 +35248,6 @@ label_replace(sum by (rpc_method, rpc_connect_rpc_error_code) (rate(rpc_client_d
-## Entitlements usage history telemetry - -

Monitoring entitlement usage history telemetry export and queue health in Sourcegraph.

- -To see this dashboard, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history` on your Sourcegraph instance. - -### Entitlements usage history telemetry: Entitlements usage history telemetry: events export and queue metrics - -#### entitlements-usage-history: entitlements_usage_history_telemetry_exported_events - -

Entitlement usage history events exported per hour

- -The number of entitlement usage history events exported as telemetry over the last hour. - -This panel has no related alerts. - -To see this panel, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history?viewPanel=100000` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -sum(increase(src_entitlementstelemetry_usage_history_exported_events[1h])) -``` -
- -
- -#### entitlements-usage-history: entitlements_usage_history_telemetry_batch_size - -

Number of entitlement usage history events exported per batch over 30m

- -The number of entitlement usage history events exported in each batch. The largest bucket is the maximum number of events exported per batch. -If the distribution trends to the maximum bucket, export throughput is at or approaching saturation. - -This panel has no related alerts. - -To see this panel, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history?viewPanel=100001` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -sum by (le) (rate(src_entitlementstelemetry_usage_history_batch_size_bucket[30m])) -``` -
- -
- -#### entitlements-usage-history: entitlements_usage_history_telemetry_pruned_events - -

Entitlement usage history events pruned per hour

- -The number of already-exported entitlement usage history events pruned over the last hour. - -This panel has no related alerts. - -To see this panel, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history?viewPanel=100010` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -sum(increase(src_entitlementstelemetry_usage_history_events_pruned[1h])) -``` -
- -
- -#### entitlements-usage-history: entitlements_usage_history_telemetry_queue_size - -

Entitlement usage history events pending telemetry export

- -The number of entitlement usage history events waiting to be exported as telemetry. - -This panel has no related alerts. - -To see this panel, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history?viewPanel=100011` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -( - sum( - src_entitlementstelemetry_usage_history_queue_size - and on(instance, job) - ( - ( - increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"}[15m]) > 0 - ) - or on(instance, job) - ( - increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total{job=~"^worker.*"}[15m]) > 0 - ) - ) - ) - ) or on() vector(0) -``` -
- -
- -#### entitlements-usage-history: entitlements_usage_history_telemetry_queue_oldest_age - -

Age of the oldest unexported entitlement usage history event

- -The age in seconds of the oldest entitlement usage history event waiting to be exported as telemetry. - -This panel has no related alerts. - -To see this panel, visit `/-/debug/grafana/d/entitlements-usage-history/entitlements-usage-history?viewPanel=100020` on your Sourcegraph instance. - -*Managed by the Sourcegraph Services team.* - -
-Technical details - -Query: - -``` -( - max( - src_entitlementstelemetry_usage_history_queue_oldest_age_seconds - and on(instance, job) - ( - ( - increase(src_entitlementstelemetry_usage_history_queue_metrics_total{job=~"^worker.*"}[15m]) > 0 - ) - or on(instance, job) - ( - increase(src_entitlementstelemetry_usage_history_queue_metrics_errors_total{job=~"^worker.*"}[15m]) > 0 - ) - ) - ) - ) or on() vector(0) -``` -
- -
- ## OpenTelemetry Collector

The OpenTelemetry collector ingests OpenTelemetry data from Sourcegraph and exports it to the configured backends.

@@ -35692,6 +35708,134 @@ sum by (code)(irate(src_http_request_duration_seconds_count{route=~"^cody.comple
+## Agent tool calls + +

Shared tool execution across MCP, Deep Search, Batch Change Agents, and other agent workloads.

+ +To see this dashboard, visit `/-/debug/grafana/d/agent-tools/agent-tools` on your Sourcegraph instance. + +### Agent tool calls: Tool calls + +#### agent-tools: agent_tool_call_rate + +

Tool calls per minute by workload, tool, and terminal status

+ +Actual executions of registered agent tools. Workload identifies the product entrypoint; status distinguishes successful calls, errors, Sourcegraph-observed timeouts, and cancellations. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/agent-tools/agent-tools?viewPanel=100000` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, tool, status) (rate(src_agent_tool_calls_total{workload=~"$workload",tool=~"$tool"}[5m])) * 60 +``` +
+ +
+ +#### agent-tools: agent_tool_non_success_rate + +

Percentage of tool calls not completing successfully over 5m by workload and tool

+ +Share of each workload and tool`s calls ending in error, timeout, or cancellation. Compare the same tool across workloads to separate tool behavior from entrypoint-specific behavior. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/agent-tools/agent-tools?viewPanel=100001` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, tool) (rate(src_agent_tool_calls_total{workload=~"$workload",tool=~"$tool",status!="success"}[5m])) / sum by (workload, tool) (rate(src_agent_tool_calls_total{workload=~"$workload",tool=~"$tool"}[5m])) * 100 +``` +
+ +
+ +#### agent-tools: agent_tool_p50_duration + +

Median tool call duration over 5m by workload, tool, and terminal status

+ +Typical execution time. The shared histogram has fine sub-second buckets for normally fast tools and extends through 20 minutes for intentionally slow tools. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/agent-tools/agent-tools?viewPanel=100010` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.50, sum by (le, workload, tool, status) (rate(src_agent_tool_call_duration_seconds_bucket{workload=~"$workload",tool=~"$tool"}[5m]))) +``` +
+ +
+ +#### agent-tools: agent_tool_p95_duration + +

95th percentile tool call duration over 5m by workload, tool, and terminal status

+ +Tail execution time. Buckets at 29s, 30s, 31s, 45s, 60s, 90s, and 120s expose common timeout boundaries, while longer buckets cover Deep Search and wait-style tools. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/agent-tools/agent-tools?viewPanel=100011` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +histogram_quantile(0.95, sum by (le, workload, tool, status) (rate(src_agent_tool_call_duration_seconds_bucket{workload=~"$workload",tool=~"$tool"}[5m]))) +``` +
+ +
+ +#### agent-tools: agent_tool_timeout_and_cancellation_count + +

Tool calls timing out or being canceled every 5m

+ +Raw timeout and cancellation counts. A sharp duration boundary in canceled calls usually indicates a client or upstream deadline; timeout means the tool returned or observed a deadline exceeded error. + +This panel has no related alerts. + +To see this panel, visit `/-/debug/grafana/d/agent-tools/agent-tools?viewPanel=100020` on your Sourcegraph instance. + +*Managed by the Sourcegraph Code Understanding team.* + +
+Technical details + +Query: + +``` +sum by (workload, tool, status) (increase(src_agent_tool_calls_total{workload=~"$workload",tool=~"$tool",status=~"timeout|canceled"}[5m])) +``` +
+ +
+ ## Deep Search

Monitoring for Deep Search question processing.

@@ -38622,4 +38766,4 @@ sum by (domain) (idelta(src_workerutil_queue_depth[10m])) / 600 ``` -
+
\ No newline at end of file