diff --git a/AGENTS.md b/AGENTS.md index 6c4c4cac..97eedcf8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,17 +2,81 @@ Welcome, AI Agent! This file contains the essential context, standards, and practices for working on this repository. Please read and adhere strictly to these guidelines before making any changes. +The single most important thing you can do here is **write documentation that sounds like a knowledgeable person wrote it for another person**. The technical rules below keep the build green; Section 2 keeps the docs worth reading. Treat both as equally required. + ## 1. Project Context + - **Product:** KloudMate is an end-to-end observability platform providing unified visibility into logs, metrics, traces, and alerts across cloud environments. - **Repository Purpose:** This repository houses the public-facing documentation sites, split into three main areas: Platform Docs, Guides, and API Docs. The site will be hosted on GitHub Pages with a custom domain (`docs.kloudmate.com`). -- **Voice & Tone:** Professional, clear, concise, and instructional. +- **Audience:** Mostly engineers and DevOps practitioners who are mid-task and looking for the fastest reliable path to "it works." Write for that person, not for a marketing landing page. +- **Voice & Tone:** Professional, clear, concise, and instructional — see Section 2 for what that actually means in practice. + +## 2. Writing Style & Voice (READ THIS BEFORE WRITING CONTENT) + +Good documentation reads like a knowledgeable colleague explaining something at your desk — not like a manual, and definitely not like ad copy. Picture the reader as someone who is in the middle of a task, slightly frustrated, and scanning for the answer. Everything below serves that reader. + +### Write to the reader, about their task + +- Use second person ("you"). Address the reader directly: "You'll need an API key before you start," not "Users must first obtain an API key." +- Make the reader the subject, not the product. Write "you land on the dashboard," not "KloudMate lands you on the dashboard." Naming the product once is fine and sometimes useful, but when it's the grammatical subject of sentence after sentence, the page reads like a feature announcement instead of a guide. The reader is the one doing the task — keep them at the center. When there's no natural "you," describe the system state instead ("the **Setup checklist** drawer opens automatically"). +- Lead with the goal, then the steps. Open a page by saying what the reader will accomplish and when they'd use it — not with history or background they didn't ask for. +- Prefer active voice and present tense. "The alert fires within seconds" beats "An alert will be sent." +- Cut the throat-clearing. Phrases like "It's worth noting that…" or "In this section, we will discuss…" add nothing. Just say the thing. + +### Sound like a person, not a model + +- Vary your sentence length. A run of identical medium-length sentences is the clearest sign a machine wrote the page. Mix short, direct lines with longer explanatory ones. +- Watch for repeated sentence openers, too. Three sentences in a row that start with the same word — especially the product name — turn into a drumbeat. Recast some of them so the reader, an action, or the result leads instead. +- Contractions are good and usually better — "don't," "you'll," "it's." They match how people actually talk. +- Be concrete. Show a real endpoint, a real value, a real error message. "Set `retention_days` to `30`" is more useful than "configure the retention setting appropriately." +- Don't pad. If a sentence can be deleted without losing meaning, delete it. + +### Words and phrases to avoid + +These read as filler or AI boilerplate. Strike them on sight: + +- **Hype:** powerful, robust, seamless, cutting-edge, blazing-fast, world-class, game-changing, effortless, supercharge. +- **Filler openers:** "In today's fast-paced world," "It's worth noting that," "As we all know," "Needless to say," "At the end of the day." +- **Overused verbs:** *leverage* → use "use"; *utilize* → use "use"; *delve into* → use "cover" or "explain"; *facilitate* → use "let" or "help." +- **Condescending qualifiers:** simply, just, easy, obviously, of course. If a step were genuinely simple, the reader wouldn't be on this page. Telling a stuck user that something is "easy" only makes them feel worse. +- **Vague intensifiers:** very, really, quite, "a variety of," "a wide range of," "a number of." + +### Be honest and helpful about the hard parts + +- Call out gotchas, prerequisites, and common mistakes *where the reader will hit them* — not in a buried footnote at the bottom. +- If something is genuinely tricky or has a known limitation, say so. Pretending everything is frictionless costs you the reader's trust the moment they hit the friction. +- Use Starlight's asides (`:::note`, `:::tip`, `:::caution`) for these, but sparingly. Overusing callouts trains readers to ignore them. + +### Structure for scanning + +- Most readers scan before they read. Use descriptive headings that say what the section covers ("Configure log retention") rather than generic or clever ones ("Configuration", "The fun part"). +- Keep paragraphs short — two to four sentences. One idea per paragraph; if a paragraph turns a corner into a new topic, start a new one. +- Use numbered lists for ordered procedures and bullets only for genuinely unordered items. Don't bullet-point prose that flows better as a sentence. + +### Make every code example runnable + +- Examples should work if copied and pasted. Always label what the reader must swap out, and prefer a concrete placeholder (`YOUR_API_KEY`) over a vague one. +- Show the expected output or success state when it helps the reader confirm they did it right. +- Keep examples minimal — just enough to make the point, nothing decorative. + +### A quick before / after + +> **Avoid:** +> "KloudMate offers a powerful and seamless way to effortlessly leverage your observability data. In today's complex cloud environments, it's worth noting that users can simply configure a wide variety of robust alerting options to suit their needs." +> +> **Prefer:** +> "Alerts tell you when something needs attention — an error-rate spike, a service going quiet, a budget threshold crossed. This page walks you through creating your first alert and routing it to Slack." + +The "prefer" version is shorter, says what the reader gets, names real scenarios, and keeps the reader (not the product) at the center. Aim for that on every page. + +## 3. Tech Stack & Architecture -## 2. Tech Stack & Architecture - **Framework:** [Astro](https://astro.build/) - **Theme:** [Starlight](https://starlight.astro.build/) - **Content Format:** Markdown (`.md`) and MDX (`.mdx`) -## 3. Directory Structure & Routing (CRITICAL) +## 4. Directory Structure & Routing (CRITICAL) + The site is divided into three distinct sections, each with its own path and navigation. **Do not mix content between these directories.** - **Platform Docs (`/docs`)**: Resides in `src/content/docs/docs/` @@ -21,14 +85,12 @@ The site is divided into three distinct sections, each with its own path and nav *Note: Any assets (images) related to these sections should be placed logically near the content or in a dedicated `public/` or `src/assets/` directory following existing patterns.* -## 4. Navigation & Sidebar Management -Each section has its own independent sidebar navigation. Whenever you create, move, or delete a file, you **MUST** update the `sidebar` configuration in `astro.config.mjs`. -- Starlight supports defining different sidebars based on the route. Ensure you are updating the correct sidebar section for the content you are working on (e.g., the sidebar for `/docs` vs the sidebar for `/guides`). - -## 5. Strict Content Rules (MUST FOLLOW) +## 6. Strict Content Rules (MUST FOLLOW) ### Frontmatter Requirements -Every hand-written `.md` and `.mdx` file **must** include `title` and `description` in its frontmatter. This is strictly enforced by a validation script. + +Every hand-written `.md` and `.mdx` file **must** include `title` and `description` in its frontmatter. This is strictly enforced by a validation script. Write the `description` like a real summary a person would skim in search results — one plain sentence, no keyword stuffing. + ```yaml --- title: Your Page Title @@ -37,33 +99,51 @@ description: A brief summary of what this page covers. ``` ### Markdown Linting -The project uses `markdownlint-cli2`. Custom rules are defined in `.markdownlint.json`. + +The project uses `markdownlint-cli2`. Custom rules are defined in `.markdownlint.json`. + - **Note:** Inline HTML is permitted (`MD033: false`), and line lengths are not strictly enforced (`MD013: false`). Follow existing file patterns. -### Cross-Linking & References -- When linking between sections or pages, ensure you use strictly relative paths (e.g., `./` or `../`). -- **CRITICAL:** Do NOT use root-relative paths starting with a forward slash (like `/docs/getting-started/`) because the `starlight-links-validator` strictly enforces link checking based on the configured deployment `base` path (e.g., in `astro.config.mjs`). Root-relative paths will break the build during the `astro:build:done` hook. -- Avoid hardcoding absolute domain URLs (like `https://docs.kloudmate.com`); use relative paths so local testing and GitHub Pages deployments work seamlessly. +## 7. API Documentation Rules -## 6. API Documentation Rules -API documentation is intended to be generated from OpenAPI/Swagger specifications rather than hand-written to avoid errors and save time. -- **Agents:** Look for an OpenAPI spec file (e.g., `openapi.yaml` or `openapi.json`) and relevant plugins in `astro.config.mjs` (like `starlight-openapi`). If updating API docs, update the source spec file, **never** the generated `.md`/`.mdx` files. +API documentation is intended to be generated from OpenAPI/Swagger specifications rather than hand-written to avoid errors and save time. -## 7. Configuration & Deployment Notes -- **GitHub Pages:** The site is configured for deployment on GitHub Pages. Be aware that the `base` property in `astro.config.mjs` might need adjustments depending on whether it's served from a root custom domain or a subdirectory. Always verify routing behaviors if you alter the `base` configuration. +- **Agents:** Look for an OpenAPI spec file (e.g., `openapi.yaml` or `openapi.json`) and relevant plugins in `astro.config.mjs` (like `starlight-openapi`). If updating API docs, update the source spec file, **never** the generated `.md`/`.mdx` files. +- The writing rules in Section 2 still apply to the parts you *do* hand-write — endpoint summaries, descriptions, and field docs in the spec. Keep them concrete and plain. ## 8. Agent Workflow & Verification Commands + Always verify your changes by running the corresponding checks before concluding your task: -1. **Check Frontmatter Validity:** +1. **Check Frontmatter Validity:** + ```bash npm run validate:frontmatter ``` + 2. **Lint Markdown Files:** + ```bash npm run lint:md ``` + 3. **Build the Site (Catches Broken Links):** Verify there are no Astro compilation or broken link errors. This is critical for catching bad relative links across the different documentation sections. + ```bash npm run build - ``` \ No newline at end of file + npm run check:links + ``` + +## 9. Self-Review Checklist (before you finish) + +Run through this on any page you wrote or edited. It's the human-quality equivalent of the build checks above. + +- [ ] Does the page open by telling the reader what they'll accomplish? +- [ ] Did I read it out loud (or in my head)? Does it sound like a person, or like a template? +- [ ] Is the reader ("you") the subject of most sentences, rather than the product? Does any product name appear as the subject several sentences in a row? +- [ ] Are there any banned words from Section 2 (powerful, seamless, leverage, simply, just…)? +- [ ] Do the sentences vary in length, or do they all march at the same pace? +- [ ] Will every code block actually run if pasted? +- [ ] Are gotchas placed where the reader hits them, not buried at the end? +- [ ] Could I delete any sentence without losing meaning? If so, delete it. +- [ ] Do `validate:frontmatter`, `lint:md`, and `build` all pass? \ No newline at end of file diff --git a/astro.config.mjs b/astro.config.mjs index 114da3a2..7e655f26 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -62,6 +62,7 @@ export default defineConfig({ ], title: 'KloudMate Docs', description: 'Documentation for KloudMate observability platform', + favicon: '/favicon.ico', logo: { light: './src/assets/logo-light.png', dark: './src/assets/logo-dark.png', diff --git a/docs/incident-management-v2-docs-plan.md b/docs/incident-management-v2-docs-plan.md new file mode 100644 index 00000000..24eb3c22 --- /dev/null +++ b/docs/incident-management-v2-docs-plan.md @@ -0,0 +1,372 @@ +# Incident Management v2 — Documentation Plan & Handoff + +**Status:** Ready for docs execution +**Owner of this brief:** Engineering (handoff to docs) +**Last updated:** 2026-06-01 + +This is an internal planning/handoff document (it lives in `docs/`, not in `src/content/docs/`, so it is **not** published). It tells whoever writes the docs exactly which pages to create or update for the Incident Management (IM) v2 redesign, and points each one at the source code so the writing stays grounded in real behavior. Read Section 2 before touching any page. + +--- + +## 1. What shipped in Incident Management v2 + +IM grew from a basic "alerts → incidents → escalation" flow into a full incident-response suite. Three features are brand new; the rest were reworked. + +**New surfaces (no docs exist yet):** + +- **On-Call Schedules** — layered rotations with daily/weekly/custom cadence, time restrictions, overrides, a live rotation preview, and a "who's on call now" view. *This is the largest and most complex addition — see the dedicated deep-dive in Section 5.* +- **Routing Rules (IM)** — a rule engine that decides, when an alert reaches IM, which service/escalation policy/severity an incident gets, whether to suppress it, and who to add or notify. **Not the same feature as the existing `alerts/routing-rules.mdx`** — see the disambiguation in Section 2.4. +- **Status Pages** — branded public status pages (components, incident/maintenance updates, 90-day uptime history, RSS/Atom), custom domains with DNS verification, and one-click "publish this incident to a status page." + +**Reworked surfaces (docs exist, need updating):** + +- **Incidents** — new activity **timeline** that merges lifecycle events and per-channel notification deliveries; responders; severity/summary editing; bulk acknowledge/resolve; incident **number** + **key**; "publish to status page." +- **Escalation Policies** — steps now target **users, on-call schedules, and Slack channels**; per-step notify methods (email/SMS/voice); retries on no-acknowledgement (ack timeout + max retries); policy-level initial wait and repeat count. +- **Alert Sources (Integrations)** — the nav now calls this **"Alert Sources"**; webhook integrations gained a **Liquid templating** system with a live template tester and sample payloads. +- **Services** — owner, default escalation policy, and SLA target. +- **Slack** — single app for both notifications and ChatOps (minor changes). + +**Plan gating:** On-Call Schedules, Routing Rules, and Status Pages are gated behind workspace capabilities — see `kloudmate-frontend/src/hooks/useImFeature.ts`, which exposes `canUseOncall`, `canUseRoutingRules`, `canUseStatusPages`, plus `canUseSms`/`canUseVoice` for the SMS/voice notify methods. Treat them as paid/Pro features and add a short availability `:::note` at the top of each gated page. Confirm the exact plan names with product before publishing. + +--- + +## 2. Before you write: grounding and conventions + +### 2.1 Source code — where to verify behavior + +Two repos, two feature branches. Always check the branch, not `develop`/`main`. + +| Repo | Path | Branch with this work | +|---|---|---| +| Frontend | `/Users/amitava/work/kloudmate-frontend` | `feat/incident-v2` | +| Backend | `/Users/amitava/work/kloudmate-backend` | `feat/oncall-v2` | + +Frontend IM module root: `src/modules/im/`. Backend IM roots: `src/routes/incident-management.ts`, `src/routes/status-page-public.ts`, `src/functions/incident-management/`, `src/services/incident-management/`, `src/dal/`. + +Every page section below lists the **exact files** to read before writing it. Enumerated values (severities, statuses, rotation types, etc.) should be copied from the constants/model files in Section 6, never invented. + +### 2.2 Authoring conventions (from `AGENTS.md`) + +- **Format:** `.mdx`. Don't add a manual `# H1` — the `title` frontmatter renders it. Start body headings at `##`. +- **Required frontmatter:** `title` and `description` (enforced by `npm run validate:frontmatter`). Use `sidebar.label` and `sidebar.order` to control the sidebar. +- **Drafts:** add `draft: true` to keep a page out of the production build while it's missing screenshots or under review. (The existing `integrating-with-kloudmate-alerts.mdx` and `alerts/routing-rules.mdx` already use this.) +- **Folder ordering:** each section folder has a `_meta.json` like `{ "label": "...", "order": N, "collapsed": true }`. Sidebar order = folder `_meta.json` order for folders, `sidebar.order` frontmatter for leaf pages. **New folders need a `_meta.json`.** +- **Components:** `import { Steps, Tabs, TabItem, LinkCard, CardGrid } from '@astrojs/starlight/components';`. Use `` for procedures, `:::note` / `:::tip` / `:::caution` for asides (sparingly). +- **Voice:** second person, active voice, concrete values, varied sentence length. Avoid the banned words list (powerful, seamless, leverage, simply, just, robust, …). Run the Section 9 self-review. +- **Validation before done:** `npm run validate:frontmatter`, `npm run lint:md`, `npm run build`, `npm run check:links`. + +### 2.3 Images and placeholders + +- Real screenshots: store next to the page in an `images/` subfolder, reference with `![alt](./images/.png)`. +- **Placeholders (use these everywhere a screenshot is needed for now):** a shared placeholder image already exists at `src/content/docs/docs/_placeholders/screenshot-placeholder.png`. Follow the pattern already used in `integrating-with-kloudmate-alerts.mdx`: + + ```mdx + :::note[Placeholder image] + Screenshot pending — . + ::: + ![](../../_placeholders/screenshot-placeholder.png) + ``` + +- **Relative depth matters.** The placeholder lives in `docs/_placeholders/`. From a page inside an IM **subfolder** (e.g. `incident-management/oncall/overview.mdx`) the path is `../../_placeholders/screenshot-placeholder.png`. From a page directly under `incident-management/` (e.g. `what-is-incident-management.mdx`) it is `../_placeholders/screenshot-placeholder.png`. Verify with `npm run build` — broken image links fail the build. +- Every screenshot needed across this work is listed in the Section 7 checklist so nothing is missed when real captures arrive. + +### 2.4 Terminology decisions to make first + +These three naming issues will cause reader confusion if not handled deliberately. Decide each with product before writing. + +1. **"Integrations" → "Alert Sources".** The product left-nav (`src/components/Sidebar/IncidentsSubMenu.js:19`) labels the integrations section **"Alert Sources"**, though the URL is still `/im/integrations`. Existing docs say "Integrations." **Recommendation:** rename the docs section heading/label to **"Alert Sources"** to match the UI, keep the folder name `integrations/`, and add a one-line "(formerly Integrations)" note in the overview. Update cross-links accordingly. + +2. **Two different "Routing Rules."** There are two unrelated features with the same name: + - **Alerts → Routing Rules** (`src/content/docs/docs/alerts/routing-rules.mdx`, already drafted): routes *alert notifications* to channels using label matchers, group-by, and cadence. Part of the alerting/notification-policy system. + - **IM → Routing Rules** (NEW, `src/modules/im/containers/routing-rules/`): routes an *incoming alert into an incident* — picks the service, escalation policy, severity; can suppress, add responders, or notify on-call. Part of the incident pipeline. + + These are genuinely different. **Do not merge them.** The new IM page (Section 4.7) must open with a one-paragraph "this is incident routing inside IM, not alert-notification routing — see [Alerts → Routing Rules] for that" disambiguation, and the alerts page should get the reciprocal note. Mirror the product's own labels. + +3. **"Alarm" → "Alert."** A recent docs commit renamed alarm → alert across the site. Keep new IM copy on **"alert"** and **"Alert Sources,"** but note that integration *types* in code still use the value `cloudwatch`/`kloudmate` and the UI labels are "CloudWatch Alarm" / "KloudMate Alarm" (Section 6). Document the labels as the user sees them. + +--- + +## 3. Information architecture — proposed sidebar + +### 3.1 Mirror the product nav + +The in-product IM nav order (`src/components/Sidebar/IncidentsSubMenu.js`) is the canonical order users already know: + +`Overview · Incidents · Services · Alert Sources · Escalation Policies · On-Call Schedules · Routing Rules · Status Pages · Settings` + +Align the docs sidebar with this. Keep the two conceptual pages (**What Is IM?**, **Incident Lifecycle**) near the top because new readers need them before the feature pages. + +### 3.2 Current vs proposed order (`incident-management/`) + +| Order | Page / folder | Now | Action | +|---|---|---|---| +| 1 | `index.mdx` (Overview) | exists | **Update** (add new sections to the lists) | +| 2 | `what-is-incident-management.mdx` | exists | **Update** | +| 3 | `incident-lifecycle.mdx` | exists | **Update** (add routing rules + on-call to the flow) | +| 4 | `incidents/` | exists (order 4) | **Update** | +| 5 | `services/` | exists (order 5) | **Update** | +| 6 | `integrations/` → relabel **Alert Sources** | exists (order 7) | **Update** + change `_meta.json` order to 6, label to "Alert Sources" | +| 7 | `escalation-policy/` | exists (order 6) | **Update** + change `_meta.json` order to 7 | +| 8 | `oncall/` | — | **Create** (new folder, `_meta.json` order 8) | +| 9 | `routing-rules/` | — | **Create** (new folder, `_meta.json` order 9) | +| 10 | `status-pages/` | — | **Create** (new folder, `_meta.json` order 10) | +| 11 | `slack-integration.mdx` | exists (order 8) | **Update** + change `sidebar.order` to 11 | + +The only existing `_meta.json`/order edits required: swap Alert Sources (7→6) and Escalation Policies (6→7), bump Slack to 11, and add three new folder `_meta.json` files. New `_meta.json` templates: + +```json +// incident-management/oncall/_meta.json +{ "label": "On-Call Schedules", "order": 8 } +``` +```json +// incident-management/routing-rules/_meta.json +{ "label": "Routing Rules", "order": 9 } +``` +```json +// incident-management/status-pages/_meta.json +{ "label": "Status Pages", "order": 10 } +``` + +--- + +## 4. Page-by-page work plan + +Each entry gives the action, file, what to cover, the **grounding files** to read, and the screenshots to placeholder. Reuse the existing page patterns (`incidents/creating-an-incident.mdx`, etc.) for tone and structure. + +### 4.1 Overview & conceptual pages — UPDATE + +**`incident-management/index.mdx`** — add the new sections to both lists. +- "In This Section": add **On-Call Schedules**, **Routing Rules**, **Status Pages**; rename **Integrations → Alert Sources**. +- Keep "Related Resources" but add a line distinguishing IM Routing Rules from Alerts Routing Rules. + +**`incident-management/what-is-incident-management.mdx`** — extend "Key Components" to cover On-Call, Routing Rules, and Status Pages in one or two sentences each. Frame the full pipeline: *alert source → routing rule → service → escalation policy → on-call → incident → (optional) status page.* + +**`incident-management/incident-lifecycle.mdx`** — the lifecycle diagram and prose currently go alert → integration → incident → service → escalation. Insert the **routing-rule** decision step (between alert source and incident) and the **on-call resolution** step (escalation step targets a schedule → schedule resolves to a person). Update the diagram image (placeholder for now). Add a short "Where on-call fits" and "Where routing rules fit" paragraph. +- **Grounding:** `kloudmate-backend/src/services/incident-management/integrations/webhook.ts` (the ingestion→routing→incident→escalation pipeline), Section 5 of this doc. + +### 4.2 Incidents — UPDATE + +**`incidents/index.mdx`** — statuses are still **Triggered → Acknowledged → Resolved**, but add: bulk acknowledge/resolve and bulk severity change from the list; filters by service, alert source, escalation policy, and "My incidents"; the incident **number** and **key** columns. +- **Grounding (frontend):** `src/modules/im/containers/incidents/List.js`, `src/modules/im/components/incidents/IncidentsTable.js`. + +**`incidents/incident-details.mdx`** — this is the biggest incidents change. Cover: +- The **activity timeline** (the headline new feature): it merges *lifecycle events* (Created, Acknowledged, UnAcknowledged, Resolved, UnResolved, Assigned, Unassigned, SeverityChanged, SummaryUpdated) with *notification delivery events* (per channel: email/SMS/voice/Slack/Teams/webhook/Jira/SNS; per status: queued/sent/delivered/failed/skipped_unverified/skipped_no_credits), including step order, retry attempt, and fallback order. Explain how to read it as a single audit trail. +- **Responders** (add/remove), **severity** edit, **summary** edit. +- **Alarms** tab — the raw alert payloads that fed the incident. +- **Notes** tab. +- **"Publish to status page"** — link to the Status Pages page; this opens the publish modal. +- **Grounding (frontend):** `src/modules/im/containers/incidents/Details.js`; `src/modules/im/components/incidents/` → `IncidentLog.js` (event + delivery rendering and the channel/status enums), `incidentTimeline.js` (merge + sort logic), `IncidentNotes.js`, `AddRespondersModal.js`, `EditSeverityModal.js`, `EditSummaryModal.js`, `IncidentAlarms.js`, `PublishToStatusPageModal` (under `status-pages/`). +- **Grounding (backend):** incident model + `incident_logs`, `escalation_step_deliveries`, `incident_responders` in `kloudmate-backend/src/dal/`; actions handled at `routes/incident-management.ts` (`POST /incidents/:id/:action` → acknowledge/resolve/unacknowledge/unresolve). +- **Screenshots:** timeline with mixed lifecycle + delivery rows; responders panel; publish-to-status-page modal. + +### 4.3 Services — UPDATE + +**`services/index.mdx`** and **`services/creating-services.mdx`** — service now carries **owner**, **default escalation policy** (required), and **SLA target** (0–100%). The create form has: name, description (required), owner, escalation policy (with an inline "create new escalation policy" link), SLA target. Detail view lists the service's alert sources and links the escalation policy. +- **Grounding (frontend):** `src/modules/im/components/services/ServiceForm.js`, `src/modules/im/containers/services/ServiceDetails.js`, `List.js`. +- **Grounding (backend):** `im_service` model (`owner_id`, `escalation_policy_id`, `sla_target`) in `src/dal/im_service/`. +- **Screenshots:** updated create form (owner + SLA fields); service detail with alert sources table. + +### 4.4 Alert Sources (Integrations) — UPDATE + relabel + +**`integrations/_meta.json`** — `label` → `"Alert Sources"`, `order` → `6`. + +**`integrations/index.mdx`** — relabel to Alert Sources; add the "(formerly Integrations)" note. Types: **CloudWatch Alarm**, **KloudMate Alarm**, **Webhook**. Each alert source maps an inbound payload to incident fields, routes to a **service** (required), and may set a default **escalation policy** (optional). + +**`integrations/adding-integrations.mdx`** — expand the Webhook section with the new **templating system**: +- Liquid templates for **Title**, **Description**, **Severity**, **Grouping**, and **Auto-Resolve**. +- The **template tester**: pick a sample payload (CloudWatch alarm or KloudMate alarm), edit a payload, run the template, see rendered output, and open the syntax reference (variables `{{ field }}`, conditions `{% if %}`, loops `{% for %}`, filters like `| raw`). +- Note the inbound webhook URL is generated per source (a hashed `/hooks/:hash` endpoint) and CloudWatch uses SNS. +- **Grounding (frontend):** `src/modules/im/components/Integrations/IntegrationForm.js`, `TemplateTester.js`, `templateValidation.js`, and `sample-payloads/cw-alarm.json`, `km-alarm.json`. +- **Grounding (backend):** `src/services/incident-management/integrations/webhook.ts` (ingestion), `mapping.ts` (Liquid → fields); `im_integrations` model (`type`, `hash`, `config`, `service_id`, `escalation_policy_id`) in `src/dal/im_integrations/`; route `POST /hooks/:hash`. +- **Screenshots:** webhook form with template fields; template tester with a rendered result; syntax reference dialog. + +**`integrations/integrating-with-kloudmate-alerts.mdx`** (currently `draft: true`) — revisit after the IM Routing Rules page (4.7) exists. This page describes wiring KloudMate alerts into IM via a notification channel + the **alerts-side** routing rule. Make sure its "Routing Rule" references clearly point at `alerts/routing-rules/` and add a "see also IM Routing Rules" cross-link so readers don't confuse the two. Keep it draft until its screenshots land. + +### 4.5 Escalation Policies — UPDATE (substantial) + +**`escalation-policy/index.mdx`** and **`escalation-policy/adding-escalation-policy.mdx`** — rewrite around the new model: +- **Policy** = name, description, an initial **wait time** (minutes before step 1), and a **repeat count** (how many times to loop the whole policy if nobody acknowledges). +- **Step** = one or more **recipients** plus an **escalate-after** delay (minutes before the next step fires). Recipients can be: + - **Users** — each with notify methods **email / SMS / voice** (SMS and voice are plan- and phone-verification-gated; the UI disables them with a tooltip when unavailable). + - **On-call schedules** — the step pages whoever is on call at fire time (gated by `canUseOncall`; UI says "On-call schedules require the Pro plan"). + - **Slack channels** — with a `fallback_order`. +- **Retries within a step** — optional ack timeout (minutes) + max retries: re-notify the same recipients until acknowledged or retries run out. Call out the gotcha the UI itself warns about: if the ack timeout ≥ escalate-after, the first retry may never run. +- Explain the end-to-end timing with a concrete example (wait 0m → Step 1 pages on-call + a backup user → 5m no ack → Step 2 pages a Slack channel + manager → repeat ×1). +- **Grounding (frontend):** `src/modules/im/components/escalation-policies/` → `PolicyForm.js`, `StepEditor.js`, `StepConnector.js` (the "escalate after / repeat from step 1" connector copy), `NotifyCheckboxes.tsx`, `RePageConfig.tsx`, `SchedulePicker.tsx`, `editorModel.js` (defaults, validation, field names). Tests in `editorModel.test.js` show the exact rules. +- **Grounding (backend):** DAL modules `escalation_policies`, `escalation_policy_logs`, `escalation_policy_step_schedules`, `escalation_step_deliveries` in `src/dal/` (step users/channels live within the escalation-policy schema/service layer); execution engine `src/services/incident-management/escalation-policy/policy.service.ts` + `queue.ts` (BullMQ delays = `wait_time_minutes`, `escalate_after_minutes`, `ack_timeout_minutes`); dispatch `src/services/notifications/handlers/incidents/escalation-policy-step/started.ts`. +- **Screenshots:** policy form with wait/repeat; a step with users + on-call schedule + Slack channel; retry config; the step connector between two steps. + +### 4.6 On-Call Schedules — CREATE (new folder `oncall/`) + +This is the most important new area and the one new users struggle with most. Give it **four pages** plus the worked examples from Section 5. Add the gating note at the top of each: *Available on .* + +1. **`oncall/index.mdx`** (Overview, `sidebar.order: 1`) — what on-call schedules are, the mental model (schedule → layers → rotation → restrictions → overrides), the schedules list, "who's on call now" and the handoff countdown, and how schedules get used (escalation-policy steps and routing-rule on-call notifications target a schedule, which resolves to a person at notification time). +2. **`oncall/creating-a-schedule.mdx`** (`order: 2`) — the create flow end to end: name, **timezone** (everything is computed in this timezone), then add a layer. Walk the layer fields: **rotation type** (daily/weekly/custom), **rotation length** (custom only, 1–365 days), **handoff time**, **start date**, **participants** (ordered — order sets who takes which shift), and the **rotation preview** calendar. Lead with the simplest possible schedule (one weekly layer, three people) — that is the 80% case. +3. **`oncall/layers-and-restrictions.mdx`** (`order: 3`) — layered schedules and time restrictions. Explain **layer priority** (top layer wins when more than one covers a moment) and **restrictions** (none / time-of-day / day-of-week; windows may cross midnight). This is where follow-the-sun and "business hours primary + 24/7 backup" patterns live (Section 5 examples 2 and 4). +4. **`oncall/overrides.mdx`** (`order: 4`) — temporary coverage swaps: pick a user, a start/end, an optional reason. Overrides beat all layers for their window. Show the "cover a vacation" case and how to remove an override. + +- **Grounding (frontend):** `src/modules/im/containers/oncall/` (`List.tsx`, `Create.tsx`, `Edit.tsx`, `Details.tsx`); `src/modules/im/components/oncall/` (`ScheduleForm.tsx` — every field, default, and validation rule; `LayerEditor.tsx`, `RestrictionsEditor.tsx`, `RotationPreview.tsx`, `OverrideDialog.tsx`, `UserPicker.tsx`, `WhoIsOnCall.tsx`); `mappers.ts` + `mappers.test.ts` (the canonical mapping of form → API, including weekly handoff-day derivation); `constants.js` (`RotationTypes`, `RestrictionTypes`, `DaysOfWeek`). +- **Grounding (backend, critical):** `kloudmate-backend/src/services/incident-management/oncall/resolver.ts` — `resolveAt` (who's on call at instant T) and `previewRange` (calendar preview). The `oncall_schedules` DAL module in `src/dal/` (layers, participants, and overrides live within the on-call schema/service layer). Routes under `/oncall/schedules` in `routes/incident-management.ts` (`/whos-on-call`, `/preview`, `/overrides`). +- **Screenshots (lots — see Section 7):** schedules list with "who's on call"; create form; layer editor with rotation type; restrictions editor; rotation preview calendar; override dialog; schedule detail with current on-call + overrides list. + +Write Section 5's worked examples into pages 2–4 (don't dump them all in one place). The user specifically asked for strong examples here — they are the difference between a reader "getting" on-call and bouncing. + +### 4.7 Routing Rules (IM) — CREATE (new folder `routing-rules/`) + +One page is enough to start; split later if it grows. Gating note at top (`canUseRoutingRules`). **Open with the disambiguation from Section 2.4.** + +**`routing-rules/index.mdx`** — cover: +- What an IM routing rule does: when an alert reaches IM (from an alert source), rules decide the incident's **service**, **escalation policy**, and **severity**, whether to **suppress** it, and whether to **add responders** or **notify on-call schedules**. +- **Scope:** workspace-wide or a single service. **Priority:** lower wins. **Enabled** toggle. +- **Matcher:** mode is **All events** / **All conditions (AND)** / **Any condition (OR)**. Conditions match on **severity**, **service**, **alert source (integration)**, **tag.**, or **payload.**, with operators **eq / neq / contains / regex / in / gt / lt** (availability varies by field — see Section 6). +- **Actions** and their semantics: **suppress**, **set severity**, **assign service**, **assign escalation policy** are *first-match-wins*; **add responders** and **notify on-call schedules** *accumulate* across all matching rules. (The action's code key is still `page_oncall_schedule_ids`; the UI label is "Notify on-call schedules.") +- **Rule tester:** feed a sample severity/service/payload and see the resulting routing decision (matched rules, final service/policy/severity, suppress, responders, notified on-call schedules) before saving. +- **Resolution precedence** to state plainly: rule action → alert source default → service default (for escalation policy and severity). +- **Grounding (frontend):** `src/modules/im/containers/routing-rules/` (`List.tsx`, `Create.tsx`, `Edit.tsx`); `src/modules/im/components/routing-rules/` (`RuleForm.tsx`, `MatcherBuilder.tsx`, `Condition.tsx` — per-field operator lists, `ActionPicker.tsx` — action definitions + first-match vs accumulate, `RuleTester.tsx`, `editorModel.ts` — the **authoritative** field/operator/serialization model). **Note:** `src/modules/im/constants.js` also has a `RoutingRuleFields` list, but it is stale for v2 (it shows `time_of_day`); trust `editorModel.ts` and `Condition.tsx`. +- **Grounding (backend):** routing engine `src/services/incident-management/routing/engine.ts`; `evaluateRoutingRules` called from `integrations/webhook.ts`; `incident_routing_rules` model (`matcher`, `actions`, `priority`, `service_id`) in `src/dal/`; routes `GET/POST/PATCH/DELETE /routing-rules` and `POST /routing-rules/test`. +- **Screenshots:** rules list with priority/scope; rule editor with matcher modes + conditions; action picker; rule tester with a decision result. + +### 4.8 Status Pages — CREATE (new folder `status-pages/`) + +Three pages: managing the page, communicating incidents, and going live on a custom domain. Gating note at top (`canUseStatusPages`). + +1. **`status-pages/index.mdx`** (Overview, `order: 1`) — what a status page is and what visitors see (overall status banner, components grouped with 90-day uptime strips, active incidents, history, RSS/Atom). The editor's three tabs: **Incidents**, **Components**, **Settings**. Creating a page: name, slug, description, primary color. **Components:** name, optional group, optional **linked service** (links a component to an IM service so its status tracks the service), display order; component statuses are **operational / degraded / partial outage / major outage**. Publish/unpublish (draft vs public). +2. **`status-pages/posting-incidents.mdx`** (`order: 2`) — communicating during an incident: create a status-page incident (title, **status**: investigating/identified/monitoring/resolved; **impact**: none/minor/major/critical; affected components; public/hidden), then post **updates** over time. Also document **"Publish to status page" from an incident** (the modal pre-fills affected components from the incident's service; it is idempotent — re-publishing updates rather than duplicates). +3. **`status-pages/branding-and-custom-domain.mdx`** (`order: 3`) — branding (logo PNG/JPEG/WebP ≤2 MB ≤512 px; favicon ≤1 MB square ≤256 px; primary color; "Hide *Powered by KloudMate*" as a paid toggle) and **custom domain** setup: add a subdomain → KloudMate returns **CNAME + TXT** records → add them at your DNS provider → **Verify** → status goes pending → active; TLS is provisioned automatically. Custom domain actions are **org-owner only**. Mention the free `{slug}.{base-domain}` URL as the no-DNS option. +- **Grounding (frontend):** `src/modules/im/containers/status-pages/` (`List.tsx`, `Editor.tsx`); `src/modules/im/components/status-pages/` (all of: `PageForm.tsx`, `ComponentList.tsx`, `BrandingSection.tsx`, `LogoUploadField.tsx`, `ImageCropDialog.tsx`, `CustomDomainPanel.tsx`, `SettingsPanel.tsx`, `IncidentEditor.tsx`, `UpdateComposer.tsx`, `PublishToStatusPageModal.tsx`, `StatusBadge.tsx`, `UptimeStrip.tsx`, `StatusPagesWidget.tsx`); `statusPagesApi.ts` (every endpoint + the impact→overall-status mapping); `constants.js` (`StatusPageIncidentStatuses`, `ImpactTypes`, `ComponentStatuses`). The **public visitor view** is the separate `src/status-page-lite/` app (`StatusPage.tsx`, `IncidentPage.tsx`, `HistorySection.tsx`, `UptimeStrip.tsx`, `api.ts`) — read it to describe what visitors see, but it is internal build detail, not something to document as a feature. +- **Grounding (backend):** `src/routes/status-page-public.ts` (public `/resolve`, `/:slug`, `/history`, `/rss`, `/atom`, `/assets/:hash`, `/tls-authorize`); `src/services/incident-management/status-pages/public.ts` + `resolve.ts` (host→page resolution, custom-domain verification, caching); the `status_pages` DAL module in `src/dal/` (page fields incl. `custom_domain`, `custom_domain_status`, verification token; components, incidents, and updates live within the status-pages schema/service layer); admin routes under `/status-pages` in `routes/incident-management.ts`. +- **Note the admin/public status vocabulary mismatch:** the admin component status `degraded` is surfaced to visitors as `degraded_performance` (the public overall status enum is `operational / degraded_performance / partial_outage / major_outage`). Document the labels the relevant audience sees; don't expose internal wire values. +- **Screenshots:** pages list with health + active incidents; editor Components tab; editor Incidents tab with the update composer; Settings → Branding; Settings → Custom domain showing DNS records + verify; the public-facing status page; an uptime strip hover; an incident permalink page. + +### 4.9 Slack / ChatOps — UPDATE (light) + +**`slack-integration.mdx`** — bump `sidebar.order` to 11. Confirm against the current UI that the "single app for notifications + ChatOps," org-level connect, and per-user account linking still match; refresh the screenshot if the flow changed. Cross-link from Escalation Policies (Slack channel recipients) and Status Pages. +- **Grounding (frontend):** `src/modules/im/components/settings/chatOps/`, `src/modules/im/containers/settings/`. +- **Grounding (backend):** `src/services/incident-management/chat-ops/`. + +--- + +## 5. On-call deep-dive (write these into Section 4.6 pages) + +The user flagged on-call as the area most in need of clear docs and examples. This section is the spec to write from — concepts first, then a field reference, then four worked examples graded from simple to advanced. Verify every value against `ScheduleForm.tsx`, `mappers.ts`, and the backend `resolver.ts`. + +### 5.1 Mental model + +- A **Schedule** is a named, timezone-bound container. Everything inside is computed in the schedule's **timezone**. +- A schedule has one or more **Layers**. Each layer rotates its own ordered list of **participants** on its own cadence. Layers are **prioritized top to bottom**: when more than one layer would be on call at a moment, the **top-most layer that has someone active wins.** +- A **rotation** advances the active participant every *rotation length* at the **handoff time**, starting from the **start date**. Daily = every 1 day, weekly = every 7 days (the handoff weekday is derived from the start date), custom = every N days (1–365). +- **Restrictions** limit when a layer is "active." With no restriction the layer is always on. *Time-of-day* applies the same window every day; *day-of-week* sets per-weekday windows. Windows may cross midnight. Outside its restriction windows, a layer yields to the next layer down. +- An **Override** replaces whoever would be on call for a specific time window (optional reason). Overrides beat every layer for that window. +- **Resolution precedence:** override → layer 1 (if active) → layer 2 (if active) → … → nobody (a real, documentable outcome — if all layers are restricted out and no override covers the moment, no one is on call). + +State this precedence explicitly. The single most common confusion is expecting a lower layer to be ignored when a higher layer exists, when in fact the lower layer only takes over where the higher one is restricted out or empty. + +### 5.2 Field reference (one table, copy values from code) + +Build a table from `ScheduleForm.tsx` and `constants.js`: Schedule (name, description, timezone); Layer (name, rotation type ∈ daily/weekly/custom, rotation length 1–365 for custom, handoff time HH:MM, start date YYYY-MM-DD, restriction type ∈ none/time-of-day/day-of-week, participants ordered); Restriction window (start, end, day 0=Sun…6=Sat for day-of-week); Override (user, start, end, reason). Note the defaults the form ships with (a new layer is weekly, 7 days, 09:00 handoff, start = today, no restriction). + +### 5.3 Worked examples + +Write each as a short scenario + the exact field values + what the rotation produces. Use real names and times. + +**Example 1 — Simple weekly rotation (the 80% case).** One layer, weekly, handoff Monday 09:00, participants [Alice, Bob, Carol]. Result: Alice covers this week, Bob next, Carol the week after, then back to Alice; the handoff happens Monday at 09:00 in the schedule's timezone. Put this on `creating-a-schedule.mdx`. + +**Example 2 — Business-hours primary with a 24/7 backup (layer priority).** Layer 1 (top): weekday day-of-week restriction Mon–Fri 09:00–17:00, participants [day-shift team]. Layer 2 (bottom): no restriction, participants [on-call engineers], weekly. Result: during business hours the primary layer is on; nights and weekends fall through to the backup layer. This teaches priority + restrictions together. Put on `layers-and-restrictions.mdx`. + +**Example 3 — Override for a vacation.** Bob is on call next week but is away Wed–Thu. Add an override: user = Dana, start = Wed 00:00, end = Fri 00:00, reason "Bob PTO." Result: Dana is on call for that window only; Bob resumes Friday. Put on `overrides.mdx`. + +**Example 4 — Follow-the-sun (advanced).** Two layers using time-of-day restrictions tuned to each region's working hours (e.g. Layer 1 = APAC team 00:00–08:00 UTC, Layer 2 = EMEA 08:00–16:00 UTC, Layer 3 = Americas 16:00–24:00 UTC), each with its own participants. Result: coverage rotates around the clock by region. Emphasize the timezone caveat: restriction windows are evaluated in the **schedule's** timezone, so pick the timezone deliberately and express each region's window in that timezone. Put on `layers-and-restrictions.mdx`. + +### 5.4 How on-call connects to the rest of IM + +Close the on-call overview by showing the link: an **escalation-policy step** or a **routing-rule "page on-call" action** targets a *schedule*, and at notification time KloudMate resolves the schedule to the person on call right then (via `resolver.ts`) and pages them through their notify methods. Cross-link to Escalation Policies and Routing Rules. + +### 5.5 Gotchas to call out where readers hit them + +- Timezone drives everything (handoff time, restriction windows, start date). DST shifts are handled, but the displayed handoff can land an hour off if the reader assumed local time. +- Participant **order** decides shift assignment — reordering changes who's on call when. +- All layers restricted + no override = nobody on call. Recommend a 24/7 fallback layer for any schedule used by an escalation policy that notifies it. +- Weekly rotation's handoff **weekday** comes from the **start date** — to hand off on Mondays, pick a Monday start date. + +--- + +## 6. Enum & terminology reference (single source for writers) + +Copy these verbatim into copy; don't paraphrase the values. Frontend source: `src/modules/im/constants.js` (unless noted). + +- **Integration / Alert Source types** (`IntegrationTypes`): `cloudwatch` ("CloudWatch Alarm"), `kloudmate` ("KloudMate Alarm"), `webhook` ("Webhook"). +- **Notification methods** (`NotificationTypes`): `email`, `slack`, `sms`, `voice`. (Per-recipient notify methods in escalation steps are email/SMS/voice; SMS/voice are plan- and phone-verification-gated.) +- **Incident severity** (`SeverityTypes`): `critical`, `high`, `medium`, `low`. (Color map: critical=error, high=warning, medium=info, low=neutral.) +- **Incident status:** Triggered → Acknowledged → Resolved (derived from `acknowledged`/`resolved` flags; not a stored enum). +- **Timeline lifecycle events** (`IncidentLog.js`): Created, Acknowledged, UnAcknowledged, Resolved, UnResolved, Assigned, Unassigned, SeverityChanged, SummaryUpdated. +- **Timeline delivery channels / statuses** (`IncidentLog.js`): channels email/SMS/voice/Slack/Teams/webhook/Jira/SNS; statuses queued/sent/delivered/failed/skipped_unverified/skipped_no_credits. +- **Rotation types** (`RotationTypes`): `daily`, `weekly`, `custom`. +- **Restriction types** (`RestrictionTypes`): `none`, `time-of-day`, `day-of-week`. **Days of week** (`DaysOfWeek`): 0=Sunday … 6=Saturday. +- **Routing rule fields/operators (authoritative: `routing-rules/editorModel.ts` + `Condition.tsx`):** fields `severity`, `service_id`, `integration_id`, `tag.`, `payload.`; matcher modes `always`/`all`/`any`; operators `eq`, `neq`, `contains`, `regex`, `in`, `gt`, `lt` (which apply depends on field). Actions: `suppress`, `set_severity`, `assign_service_id`, `assign_escalation_policy_id` (first-match-wins); `add_responder_user_ids`, `page_oncall_schedule_ids` (accumulate). ⚠️ Ignore `constants.js` `RoutingRuleFields` (`time_of_day`) — stale for v2. +- **Status-page incident status** (`StatusPageIncidentStatuses`): `investigating`, `identified`, `monitoring`, `resolved`. +- **Status-page impact** (`ImpactTypes`): `none`, `minor`, `major`, `critical`. +- **Component status — admin** (`ComponentStatuses`): `operational`, `degraded`, `partial_outage`, `major_outage`. **Public overall status** (status-page-lite): `operational`, `degraded_performance`, `partial_outage`, `major_outage` (note `degraded` → `degraded_performance` on the public side). + +Backend DAL modules that confirm the above live in `kloudmate-backend/src/dal/`: `incidents`, `incident_logs`, `incident_responders`, `incident_alarms`, `incident_routing_rules`, `oncall_schedules`, `escalation_policies`, `escalation_policy_logs`, `escalation_policy_step_schedules`, `escalation_step_deliveries`, `im_integrations`, `im_service`, `status_pages`. Finer-grained entities (on-call layers/overrides, escalation step users/channels, status-page components/incidents/updates) live within those modules and the matching `src/services/incident-management/*` schema/service layers rather than as separate DAL folders. + +--- + +## 7. Screenshot / placeholder checklist + +Use the placeholder pattern (Section 2.3) for every row until real captures exist. Real images go in each section's `images/` folder; placeholders reference `_placeholders/screenshot-placeholder.png` at the correct relative depth. + +| # | Page | Screenshot shows | +|---|---|---| +| 1 | incident-lifecycle | Updated pipeline diagram incl. routing rule + on-call | +| 2 | incidents/index | List with bulk-action toolbar + filters | +| 3 | incidents/incident-details | Activity timeline with mixed lifecycle + delivery rows | +| 4 | incidents/incident-details | Responders panel | +| 5 | incidents/incident-details | Publish-to-status-page modal | +| 6 | services/creating-services | Create form with owner + SLA target | +| 7 | services/index | Service detail with alert sources table | +| 8 | integrations/adding-integrations | Webhook form with Liquid template fields | +| 9 | integrations/adding-integrations | Template tester with rendered output | +| 10 | integrations/adding-integrations | Syntax reference dialog | +| 11 | escalation-policy/adding-escalation-policy | Policy form (wait + repeat) | +| 12 | escalation-policy/adding-escalation-policy | Step with users + on-call schedule + Slack channel | +| 13 | escalation-policy/adding-escalation-policy | Retry config + step connector | +| 14 | oncall/index | Schedules list with "who's on call" + handoff countdown | +| 15 | oncall/creating-a-schedule | Create form (timezone + first layer) | +| 16 | oncall/creating-a-schedule | Layer editor with rotation type | +| 17 | oncall/creating-a-schedule | Rotation preview calendar | +| 18 | oncall/layers-and-restrictions | Restrictions editor (day-of-week windows) | +| 19 | oncall/layers-and-restrictions | Two-layer schedule (priority) | +| 20 | oncall/overrides | Override dialog | +| 21 | oncall/overrides | Schedule detail with active overrides list | +| 22 | routing-rules/index | Rules list (priority + scope) | +| 23 | routing-rules/index | Rule editor (matcher modes + conditions) | +| 24 | routing-rules/index | Action picker | +| 25 | routing-rules/index | Rule tester with decision result | +| 26 | status-pages/index | Pages list with health + active incidents | +| 27 | status-pages/index | Editor → Components tab | +| 28 | status-pages/posting-incidents | Editor → Incidents tab with update composer | +| 29 | status-pages/branding-and-custom-domain | Settings → Branding | +| 30 | status-pages/branding-and-custom-domain | Custom domain with DNS records + verify | +| 31 | status-pages/* | Public-facing status page (visitor view) | +| 32 | status-pages/* | Uptime strip hover + incident permalink | + +--- + +## 8. Suggested execution order + +1. **Conventions + IA first:** add the three new `_meta.json` files, do the order swaps (Section 3.2), and update `incident-management/index.mdx`. This makes the new sections appear in the sidebar so subsequent pages have a home. +2. **On-Call (4.6 + Section 5)** — biggest and highest-value; do it while the model is fresh. +3. **Routing Rules (4.7)** — write the disambiguation, then the page; update the reciprocal note on `alerts/routing-rules.mdx`. +4. **Status Pages (4.8).** +5. **Escalation Policies + Incidents + Services + Alert Sources updates (4.2–4.5).** +6. **Slack + lifecycle/what-is conceptual updates (4.1, 4.9).** +7. Swap real screenshots in as they arrive; drop `draft: true` per page once its captures land. + +## 9. Definition of done (per page) + +- Frontmatter valid (`title` + `description`); correct `sidebar.order`; new folders have `_meta.json`. +- Behavior matches the grounding files listed for that page (don't write from this brief alone — open the code). +- Enums/values copied from Section 6 / the constants file, not paraphrased. +- Placeholders use the shared image + a `:::note[Placeholder image]` describing the capture. +- AGENTS.md self-review passed (no banned words, reader-as-subject, varied sentences). +- `npm run validate:frontmatter`, `npm run lint:md`, `npm run build`, `npm run check:links` all green. diff --git a/public/favicon.ico b/public/favicon.ico new file mode 100644 index 00000000..0b34c569 Binary files /dev/null and b/public/favicon.ico differ diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs index 64d48f85..d638ae56 100644 --- a/scripts/check-links.mjs +++ b/scripts/check-links.mjs @@ -13,10 +13,18 @@ * page's URL and then checked against the filesystem, which is exactly what * this script does. * + * Draft pages: pages with `draft: true` frontmatter are excluded from the + * production build (via `starlight-auto-drafts`), so they never land in `dist/`. + * Links from published pages to a not-yet-published draft would otherwise be + * reported as broken. We compute the route of every draft page from source + * (mirroring the `generateId` slug rewrite in `src/content.config.ts`) and skip + * links that resolve to one — the source link stays in place and starts working + * the moment the target page drops its draft flag. + * * Usage: `node scripts/check-links.mjs` (runs automatically via `postbuild`). */ import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs'; -import { join, resolve } from 'node:path'; +import { join, resolve, relative } from 'node:path'; const distDir = resolve(process.cwd(), 'dist'); @@ -25,6 +33,51 @@ if (!existsSync(distDir)) { process.exit(1); } +// --- Draft routes ----------------------------------------------------------- +// Mirror astro.config.mjs's base normalization so computed routes match the +// resolved link pathnames (which include the configured base path). +function normalizeBasePath(pathname) { + if (!pathname || pathname === '/') return '/'; + return `/${pathname.replace(/^\/+|\/+$/g, '')}/`; +} +const base = normalizeBasePath(process.env.DOCS_BASE_PATH ?? '/'); + +const srcDir = resolve(process.cwd(), 'src/content/docs'); +const FRONTMATTER_RE = /^---\r?\n([\s\S]*?)\r?\n---/; +const DRAFT_RE = /^draft:\s*true\s*$/m; + +// Normalize a pathname for comparison: drop a trailing slash (except root). +function normalizeRoute(p) { + return p === '/' ? '/' : p.replace(/\/+$/, ''); +} + +// Convert a source file path to its built route, replicating `generateId`: +// strip extension → strip a leading `docs/` → strip a trailing `/index`. +function routeForSource(absPath) { + const rel = relative(srcDir, absPath).split('\\').join('/'); + const noExt = rel.replace(/\.(md|mdx|markdown|mdoc)$/i, ''); + const withoutDocsPrefix = noExt.replace(/^docs\//, ''); + const id = withoutDocsPrefix.replace(/\/index$/, ''); + return normalizeRoute(`${base}${id}`); +} + +const draftRoutes = new Set(); +function collectDrafts(dir) { + if (!existsSync(dir)) return; + for (const entry of readdirSync(dir)) { + const full = join(dir, entry); + const stat = statSync(full); + if (stat.isDirectory()) collectDrafts(full); + else if (/\.(md|mdx|markdown|mdoc)$/i.test(full)) { + const fm = readFileSync(full, 'utf8').match(FRONTMATTER_RE); + if (fm && DRAFT_RE.test(fm[1])) draftRoutes.add(routeForSource(full)); + } + } +} +collectDrafts(srcDir); + +let skippedDraftLinks = 0; + const issues = []; function walk(dir) { @@ -73,12 +126,21 @@ function checkFile(htmlPath) { continue; } if (targetExists(resolved)) continue; + // Skip links that point at a draft page (excluded from this build). + if (draftRoutes.has(normalizeRoute(resolved))) { + skippedDraftLinks++; + continue; + } issues.push({ page: url, href: m[1], resolved }); } } walk(distDir); +const draftNote = skippedDraftLinks + ? ` (skipped ${skippedDraftLinks} link${skippedDraftLinks === 1 ? '' : 's'} to ${draftRoutes.size} draft page${draftRoutes.size === 1 ? '' : 's'})` + : ''; + // Dedupe by (page, href) const seen = new Set(); const uniq = issues.filter((i) => { @@ -89,11 +151,11 @@ const uniq = issues.filter((i) => { }); if (uniq.length === 0) { - console.log('check-links: no broken internal links found ✓'); + console.log(`check-links: no broken internal links found ✓${draftNote}`); process.exit(0); } -console.error(`check-links: found ${uniq.length} broken internal link${uniq.length === 1 ? '' : 's'} across ${new Set(uniq.map((i) => i.page)).size} page${new Set(uniq.map((i) => i.page)).size === 1 ? '' : 's'}\n`); +console.error(`check-links: found ${uniq.length} broken internal link${uniq.length === 1 ? '' : 's'} across ${new Set(uniq.map((i) => i.page)).size} page${new Set(uniq.map((i) => i.page)).size === 1 ? '' : 's'}${draftNote}\n`); for (const i of uniq) { console.error(` ${i.page}`); console.error(` href: ${i.href}`); diff --git a/skills/starlight-docs-authoring/references/kloudmate-docs-site.md b/skills/starlight-docs-authoring/references/kloudmate-docs-site.md new file mode 100644 index 00000000..910b2f20 --- /dev/null +++ b/skills/starlight-docs-authoring/references/kloudmate-docs-site.md @@ -0,0 +1,51 @@ +# KloudMate Docs Site Reference + +## Repository Map + +- Platform docs live in `src/content/docs/docs/`. +- Guides live in `src/content/docs/guides/`. +- API reference content is generated from `openapi.yaml` through `starlight-openapi`. +- Validation commands are defined in `package.json`. + +## Rules To Prioritize + +Apply these sources in this order when they disagree: + +1. Explicit user request +2. Repository-level agent instructions such as `AGENTS.md` +3. Build and validation behavior +4. Local neighboring-file conventions +5. Generic writing guides + +This repository contains conflicting documentation guidance. Prefer the stricter rule that keeps the build green. + +## Current Repository Conventions + +- `astro.config.mjs` currently uses Starlight with autogenerated sidebar groups for `docs` and `guides`. Verify this before editing navigation logic. +- API docs are generated. Update `openapi.yaml` or the Starlight OpenAPI configuration instead of generated output. +- Hand-authored pages require `title` and `description` frontmatter. +- Neighboring content often uses `.mdx`. Match the surrounding section and do not change extensions without a reason. +- Many product docs keep screenshots in a sibling `images/` folder and link to them relatively, for example `./images/example.png`. Preserve the local pattern around the page you edit. +- Internal links should be conservative and build-safe. In this repository, prefer relative links between content pages because repo instructions are stricter than some older guide examples. + +## Authoring Checklist + +- Confirm the correct section before creating or moving a page. +- Inspect two or three nearby pages for tone, headings, images, and link patterns. +- Add or preserve `title` and `description`. +- Keep headings sequential. +- Avoid hand-editing generated API reference files. +- Check that image paths and related-page links resolve from the file being edited. +- Verify whether a navigation update is actually required. + +## Validation Checklist + +Run these commands after material content changes: + +```bash +npm run validate:frontmatter +npm run lint:md +npm run build +``` + +`npm run build` is the highest-signal check because it catches Astro compilation issues and broken links. diff --git a/src/content/docs/docs/_placeholders/screenshot-placeholder.png b/src/content/docs/docs/_placeholders/screenshot-placeholder.png new file mode 100644 index 00000000..02dd0fd2 Binary files /dev/null and b/src/content/docs/docs/_placeholders/screenshot-placeholder.png differ diff --git a/src/content/docs/docs/alarms/alarm-annotations.mdx b/src/content/docs/docs/alarms/alarm-annotations.mdx deleted file mode 100644 index 39441d10..00000000 --- a/src/content/docs/docs/alarms/alarm-annotations.mdx +++ /dev/null @@ -1,69 +0,0 @@ ---- -title: "Alarm Annotations & Severity" -description: "Learn how to use custom annotations and Liquid templating to add dynamic context and severity to your alarm notifications." -sidebar: - label: "Annotations & Severity" - order: 5 ---- - -KloudMate allows you to attach custom metadata to your alarms using **Annotations**. Annotations are key-value pairs that provide additional context to the responders receiving the notification. - -With the latest updates, annotations now support dynamic templating. This allows you to inject real-time data from the alarm evaluation directly into the notification text. - -## Setting up Annotations - -Annotations are configured in the **Alarm Details** section when creating or editing an alarm. - -1. Scroll to the **Custom Annotations** section. -2. Click **Add Annotation**. -3. Provide a **Key** (e.g., `severity`). -4. Provide a **Value**. This value can be static text or a template expression resolving the value dynamically. - -### Using Liquid Templates - -Liquid templating allows you to dynamically populate annotation values based on the query results that triggered the alarm. - -You can reference the values of queries and expressions using the `{{ }}` syntax. - -#### Common Template Variables - -- `{{ $A.value }}`: The evaluated numeric value of query/expression A. -- `{{ $B.value }}`: The evaluated numeric value of query/expression B. -- `{{ dimensions.host_name }}`: The value of the `host_name` dimension that triggered the alarm (useful for multi-dimensional alarms). -- `{{ state }}`: The current state of the alarm (e.g., `FIRING`, `NORMAL`). - -#### Example Usage - -If you have an alarm monitoring CPU utilization (Query A) grouped by `host_name`, you might create an annotation to provide immediate context in Slack: - -- **Key**: `Description` -- **Value**: `Host {{ dimensions.host_name }} is experiencing high CPU usage. Current value: {{ $A.value }}%.` - -When the alarm fires for the host `web-server-01` with a CPU of 95%, the resulting annotation in the notification will read: *Host web-server-01 is experiencing high CPU usage. Current value: 95%.* - -## Dynamic Severity - -You can use the reserved `severity` annotation key to dynamically set the severity level of the alarm based on the evaluated conditions. This is particularly useful when integrating with incident management tools like PagerDuty or Jira. - -To use dynamic severity, you typically combine Liquid templating with `if`/`else` logic based on expression values. - -### Example: Dynamic Severity based on Thresholds - -Let's assume you have an alarm evaluating an error rate metric: -- If Error Rate > 10% (0.1), it's `critical` -- If Error Rate > 5% (0.05), it's `warning` - -Create an annotation with: -- **Key**: `severity` -- **Value**: -```liquid -{% if state.value > 0.1 %} - critical -{% elsif state.value > 0.05 %} - warning -{% else %} - info -{% endif %} -``` - -When the notification is sent, KloudMate will evaluate the Liquid template and assign the appropriate severity string to the alert, allowing your downstream notification channels or incident routing rules to prioritize it correctly. diff --git a/src/content/docs/docs/alarms/create-alarms.mdx b/src/content/docs/docs/alarms/create-alarms.mdx deleted file mode 100644 index ae667ec5..00000000 --- a/src/content/docs/docs/alarms/create-alarms.mdx +++ /dev/null @@ -1,170 +0,0 @@ ---- -title: "Creating Alarms" -description: "Step-by-step guide to setting up and configuring KloudMate Alarms." -sidebar: - order: 2 ---- -KloudMate lets users create and configure alarms for events that are critical to their application. By setting up alarms, users can monitor when certain metrics cross pre-defined thresholds and take necessary actions promptly. - -## Getting Started - -Navigate to the **Alarms** section from the left navigation menu. - -![image](./images/setting-up-kloudmate-alarms-1.png) - -The **Alarms** screen displays a list of all existing alarms along with their current state, name, and description. The summary at the top shows the total number of alarm rules, including how many are currently **Firing** or **Pending**. - -From the **more options (⋯)** icon on any alarm, you can: - -- **View** the alarm details -- **View State History** for the alarm -- **Edit** the alarm configuration -- **Duplicate** the alarm -- **Pause Evaluation** or **Pause Notifications** -- **Delete** the alarm - -![image](./images/setting-up-kloudmate-alarms-1.jpeg) - -To learn about the key concepts of KloudMate Alarms, see the [Alarms Overview](../). - -## Creating a New Alarm - -Click the **Create Alarm** button at the top-right corner of the **Alarms** screen. A dialog appears with three ways to create an alarm: - -![Create Alarm dialog](./images/create-alarm-dialog.png) - -- **From Template**: Start with a pre-configured alarm for a common monitoring scenario. -- **From Scratch**: Create a custom alarm from an empty configuration. -- **Using AI**: Describe the alarm you want in plain English and let KloudMate build it for you. - -### From Template - -Instead of building an alarm from scratch, you can start from a pre-configured template that covers common monitoring scenarios and best practices. - -1. In the **Create Alarm** dialog, select **From Template**. - -2. Click the **Select a template** dropdown and choose a template that matches your monitoring needs. - -![Template selection dropdown](./images/from-template-dropdown.png) - -3. Click **Create Alarm**. The alarm is created and appears in the **Alarms** list. -4. To open and configure it, click the menu next to the alarm and select **Edit**. -5. The alarm opens pre-configured with the query, aggregation, and threshold settings from the template. Review and adjust any filters to match your environment. -6. Click **Save** or **Save & Close** when done. - -### Using AI - -KloudMate's assistant can automatically generate alarm queries and thresholds based on your natural language prompt. - -1. In the **Create Alarm** dialog, select **Using AI**. - -2. A text box appears. Describe the alarm you want to create. -3. Click **Create Alarm**. KloudMate generates the alarm configuration based on your description. -4. To review or adjust the settings, click the menu next to the alarm and select **Edit**. - -### From Scratch - -To build a fully custom alarm, select **From Scratch** in the **Create Alarm** dialog and click **Create Alarm**. - -This opens the alarm creation form where you can choose a data source, configure the metric or query to monitor, and define the alert condition on a single page. - -You can create multiple queries and expressions using the **Add Query** and **Add Expression** buttons. Each query or expression is assigned a unique alphabetical notation such as **A**, **B**, or **C**. You can duplicate any query or expression using the copy icon at the top-right corner of each block. - -To access advanced query and expression options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode** at the top of the form. - -![Advanced mode options](./images/advanced-mode-options.png) - -### 1. Setup Query Conditions and Expressions - -#### Setting Up Query Conditions for OpenTelemetry / KloudMate - -![image](./images/setting-up-kloudmate-alarms-2.png) - -- **Data Set:** Select the dataset you want to retrieve from your data source. -- **Metric to Aggregate:** Select the metric associated with the selected dataset that you want to monitor. -- **Group By:** Enter the attributes used to group the data points. -- **Filters:** Add filters to narrow down the retrieved data points. - -OpenTelemetry users can also use **Prometheus query language** to retrieve data and configure alarms. - -#### Setting Up Query Conditions for AWS (CloudWatch) - -![image](./images/setting-up-kloudmate-alarms-3.png) - -- **Time Range:** Set the duration for which data should be fetched using the dropdown, or enter a custom value in seconds. -- **Region:** Select the AWS region of the service you want to monitor. -- **Namespace:** Select the AWS service namespace you want to create an alarm for. -- **Metric:** Select the metric associated with the selected namespace. -- **Statistic:** Select the statistical function to use when calculating data points. -- **Dimensions:** Optionally configure the alarm for grouped resources within the selected namespace. For example, for EC2, you can filter by autoscaling group name, image ID, instance type, and more. - -Click **Run Query** to fetch data. - -#### Time Range Expressions for Custom Time - -Alarm query time ranges support the following: - -- **Operators:** `-` for subtracting time -- **Supported values:** The same units and keywords used in dashboards -- **Examples:** `now`, `now-5m` - -#### Setting Up Evaluation Expressions - -Expressions let you apply logic to query results. Reference any configured query or expression using its alphabetical notation, such as **A**, **B**, or **C**. An expression can be passed as a parameter only when multiple expressions are configured. - -Choose from the following expression types: - -- **Math Expression:** Enter a mathematical expression to apply to the value of a query or expression. Examples: `$A+1`, `$A<$B`, `$A && $C`. For more information, see [Alarm Expressions](../expressions/). -- **Reduce:** Select a function to aggregate the values of a query or expression into a single number, then select the target query or expression from the **Input** dropdown. Available functions include `mean()`, `max()`, `min()`, `sum()`, `last()`, and `count()`. -- **Condition Expression:** Select a function and a query or expression, then choose a condition and provide a threshold value to evaluate against. You can add multiple conditions and combine them using **AND** or **OR** logical operators. - -Click **Run Queries** to execute all configured queries and expressions. - -:::info -To avoid the **NoData** issue when using multiple queries in a single alarm, use the `ifNull` operator to assign a default value. Read more in [Alarm Expressions](../expressions/). -::: - -### 2. Configure Evaluation Settings - -![image](./images/setting-up-kloudmate-alarms-4.png) - -- **Alarm Condition:** Select the query or expression that should trigger the alarm, such as **A**, **B**, or **C**. -- **Evaluate Every:** Define how frequently the alarm condition should be evaluated, for example `1m`. -- **Pending Duration:** Define how long the alarm condition must remain true before the alarm is triggered, for example `5m`. -- **Alert State if No Data:** Select the alarm behavior when the query returns no data. -- **Alert State if Error:** Select the alarm behavior when the query returns an error. - -Click **Preview Alarms** to run the query immediately and check the result. - -### 3. Add Alarm Details - -![image](./images/setting-up-kloudmate-alarms-5.png) - -- **Alarm Name:** Enter a name for the alarm. -- **Description:** Add a description to help identify the alarm's purpose. -- **Responder Context:** Optionally add context to help on-call responders understand the alarm and act quickly. -- **Severity:** Add a free-form severity label such as `sev1`, `critical`, or `p1`. You can also use templates such as `{{ labels.* }}`, `{{ state.value }}`, and `{{ state.values.A }}`. -- **Dashboard:** Link a relevant dashboard for quick reference. -- **Summary:** Add a summary that will be included in notifications to provide context. It supports the same template variables as **Severity**. -- **Playbook URL:** Add an optional runbook or playbook URL with on-call instructions. -- **Custom Annotations:** Add custom key-value annotations. -- **SLA Target:** Set an SLA target for this alarm. - -### 4. Add Notification Tags - -Add tags to the alarm to route notifications through a matching notification policy. Each tag is a **Name/Value** pair. Click **Add tag** to add more tags. When the alarm is triggered, notifications are sent to the channels configured in the matching notification policy. - -### 5. Save the Alarm - -Click **Save** to save the alarm, or **Save & Close** to save and return to the **Alarms** screen. A confirmation message appears when the alarm is created successfully. - -## Viewing an Alarm - -To open an alarm, click the menu next to it and select **View**. This opens the alarm detail page with four tabs: - -![Alarm detail overview](./images/alarm-detail-overview.png) - -- **Overview:** Shows instance states, breaching instances with labels, reason, and duration, along with recent state transitions. -- **Instances:** Shows the full list of alarm instances and their current states. -- **History:** Shows the state change history over time. -- **Rule:** Shows the alarm configuration and query definition. diff --git a/src/content/docs/docs/alarms/index.mdx b/src/content/docs/docs/alarms/index.mdx deleted file mode 100644 index 0622cfa4..00000000 --- a/src/content/docs/docs/alarms/index.mdx +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: "Alarms Overview" -description: "Understand the core concepts of KloudMate Alarms, including rules, queries, expressions, and notification channels." -sidebar: - order: 1 - label: "Introduction" ---- -Developers create custom KloudMate alarms to monitor events critical to their applications. These alarms use a set of evaluation criteria to determine whether the alarm triggers. - -## Key Concepts - -### Alarm Rule - -An alarm rule contains evaluation criteria: one or more queries, expressions, and conditions. It specifies the evaluation frequency and the duration a condition must remain true before the alarm triggers. - -### Alarm Query - -A query provides configurational instructions to request data from a source. KloudMate alarms support AWS CloudWatch or KloudMate as data sources. - -### Alarm Expressions - -Expressions manipulate, reduce, or transform data from queries using math operations and reduction functions. They set conditions against raw query data or transformed results. - -### Notification Channel - -Notification channels are ChatOps and communication platforms that receive alarm notifications. KloudMate integrates with popular tools like Slack, Email, and more. See [Notification Channels](../platform/settings/notification-channels/) for details. - -### Notification Tags - -Tags add name-value attributes to alarms, enabling cross-referencing and mapping to notification policies. - -### Notification Policies - -Notification policies define when, where, and how to send alarm notifications. They route alerts to a specific channel when an alarm’s tags match the policy’s tags. See [Notification Policies](./notifications/) for details. - -### Alarm Labels - -Alarm labels are key-value pairs with additional information about the alarm rule. - -- KloudMate assigns labels to alarm queries based on query dimensions. -- Alarm expressions inherit labels from their source queries. -- Labels determine if operations between queries/expressions are allowed. -- Operations work if labels match exactly or one is a subset of the other. -- View labels when previewing alarms. - -### Multi-dimensional Alarms - -KloudMate supports multiple alarms per rule, called multi-dimensional alarms. For example, a rule checking Lambda throttling creates one alarm per throttled function. - -## Workflow of KloudMate Alarms - -An alarm rule: - -1. Retrieves data from the selected source using queries. -2. Reduces or transforms query results with expressions. -3. Compares queries, expressions, or thresholds using conditions. -4. Triggers only if the condition holds for the defined duration. - -Once the condition is met and remains true for the defined duration, KloudMate compares the notification tags added to the alarm against the tags specified in notification policies. When tags match exactly (all policy tags must exist on the alarm), KloudMate sends the notification to the notification channel configured in that matching policy. - -:::info -If **no tags are specified** in a notification policy, it matches all alarms regardless of their tags. -::: - -## KloudMate Alarm States - -Alarm states show current status. Filter and view alarms by state: - -- **Firing/Alerting:** Alarm active longer than configured duration. -- **Pending:** Alarm active but shorter than threshold duration. -- **Normal:** No firing or pending—everything works as expected. -- **Error:** Error occurred during evaluation. -- **No Data:** No data received for the configured time window. - -## Next Steps - -- [Creating Alarms](./create-alarms/) -- [Alarm Annotations & Severity](./alarm-annotations/) diff --git a/src/content/docs/docs/alarms/maintenance-windows.mdx b/src/content/docs/docs/alarms/maintenance-windows.mdx deleted file mode 100644 index 16a6b2d0..00000000 --- a/src/content/docs/docs/alarms/maintenance-windows.mdx +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: "Alarm Maintenance Windows" -description: "Schedule maintenance windows to suppress alarms during planned downtime." -sidebar: - order: 4 ---- -The Maintenance Window feature in the Alarm module defines a scheduled time period during which monitoring alerts are temporarily suppressed, muted, or deprioritized while planned work is performed on a system. - -## Purpose - -- It prevents false or expected alerts during deployments, patches, upgrades, or infrastructure changes. -- This avoids unnecessary escalations or incident tickets. -- It also keeps monitoring reports clean and meaningful. - -## Overview Page - -The overview page displays all created maintenance windows with these details: - -- **Name**: Name of the maintenance window. -- **Description**: Description of the maintenance window. -- **Alarm rule targeted**: Alarm rule to which the maintenance window applies. -- **Matching label**: Labels assigned to the maintenance window. -- **Schedule**: Start and end time period for the maintenance window. -- **Status**: Whether the maintenance window is active or paused. - -![image](./images/alarm-maintenance-window-1.png) - -## Creating a Maintenance Window - -1. Navigate to the **Maintenance Windows** page in the Alarm module. -2. Click **Create Maintenance Window**. - -![image](./images/alarm-maintenance-window-1.jpeg) - -3. Configure these details: - -- **Name**: Name for the maintenance window. -- **Description**: Short description of the maintenance window. -- **Timezone**: Timezone for the scheduled maintenance window. -- **One time**: A one-time maintenance window schedules a maintenance period only once for a specific date and time. -- **Recurring**: A recurring maintenance window automatically repeats on a regular cycle, so you do not need to set it up each time. - -![image](./images/alarm-maintenance-window-2.jpeg) - -Additional settings include: - -- **Start time**: Date and time when the maintenance window begins. -- **End time**: Date and time when the maintenance window ends. -- **Frequency**: Available only for recurring windows; select daily, weekly, or monthly schedules. -- **Label Matchers**: Use labels from alarms to target the maintenance window to specific services (e.g., instance name, service name). - -4. Click **Save** to create the new maintenance window. - -![image](./images/alarm-maintenance-window-3.jpeg) diff --git a/src/content/docs/docs/alarms/notifications.mdx b/src/content/docs/docs/alarms/notifications.mdx deleted file mode 100644 index bc9c1b69..00000000 --- a/src/content/docs/docs/alarms/notifications.mdx +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: "Alarm Notification Policies" -description: "Set up notification policies to receive alerts for KloudMate Alarms." -sidebar: - order: 5 ---- -To receive notifications from KloudMate about incidents in your infrastructure, you need to set up notification policies. A notification policy configures how and where you get notified for alarms or issues. - -:::tip -Before creating a policy, ensure you have configured your preferred communication platforms. See [Notification Channels](../../platform/settings/notification-channels/) to learn how to set up Slack, Email, Webhooks, Jira, and more. -::: - -:::info -In the case of multiple workspaces and AWS accounts, you must set up notifications for each account in their respective **Settings**. -::: - -## How Notifications Work - -1. The user creates KloudMate alarms. -2. The user configures their preferred notification channels such as Email, Slack, MS Teams, and more. -3. The user creates a notification policy, adding the alarm and the channel where they want to be notified. -4. When an alarm is triggered, the notification policy determines which notifications are sent via which channels. - -## Setting Up Notification Policies - -Navigate to **Settings** > **Notification Policies**. The screen displays all existing policies with their name, assigned channels, conditions, and enabled status. - -![image](./images/setting-up-notifications-3.jpeg) - -1. Click the **Create** button at the top-right corner. -2. Enter a **Policy name**. -3. Select one or more Notification channels from the dropdown. The dropdown displays channels created for the same workspace you are currently working in. - -![image](./images/setting-up-notifications-3.png) - -4. Click **Add condition** to define when this policy should trigger. You can add multiple conditions to a single policy. - -**For Alarm conditions:** -- Select the **alarm(s)** from the Select alarms dropdown. -- Optionally **add tags** (Name and Value) to match specific alarm tags. If no tags are specified, the condition will match all alarm tags. - -![image](./images/setting-up-notifications-4.png) - -**For Issue conditions:** -- Select the **Issue type** — New issues or New and regression. -- Select the **Target issue(s)** from the dropdown. - -![image](./images/setting-up-notifications-5.png) - -5. Use the **Disable this policy** toggle if you want to save the policy without activating it immediately. -6. Click **Save policy**. - -:::info -You can add multiple notification channels and multiple conditions in a single notification policy. -::: - diff --git a/src/content/docs/docs/alarms/_meta.json b/src/content/docs/docs/alerts/_meta.json similarity index 65% rename from src/content/docs/docs/alarms/_meta.json rename to src/content/docs/docs/alerts/_meta.json index 210ae37e..66d77ff4 100644 --- a/src/content/docs/docs/alarms/_meta.json +++ b/src/content/docs/docs/alerts/_meta.json @@ -1,5 +1,5 @@ { - "label": "Alarms", + "label": "Alerts", "order": 18, "collapsed": true } diff --git a/src/content/docs/docs/alerts/alert-groups.mdx b/src/content/docs/docs/alerts/alert-groups.mdx new file mode 100644 index 00000000..00903b97 --- /dev/null +++ b/src/content/docs/docs/alerts/alert-groups.mdx @@ -0,0 +1,107 @@ +--- +title: "Alert Groups" +description: "Bundle related signals into a single notifiable group with firing instances, an audit log, and dispatched notifications all in one place." +sidebar: + order: 6 +--- + +When matching alerts fire close together, KloudMate bundles them into a single **Alert Group**. Instead of paging on every individual signal, you page once on the group; new signals append to the group rather than fan out into separate notifications. + +A group survives restarts and deduplicates against the same incident over time, so the same recurring issue keeps showing up on the same group rather than spawning a fresh one each time. + +## Where to find groups + +Open **Alerts → Alert groups** in the left navigation. This is the workspace-wide view of every group, open or resolved. + +![Alert Groups list](./images/alert-groups-list.png) + +Columns: **Group** (title + label chips), **State** (Open / Resolved), **Signals**, **Opened**, **Last signal**, and an investigation indicator when [Auto-RCA](../auto-rca/) ran. + +Expand any row to preview that group's signals inline — alert name, state, severity, received time, and labels — without leaving the list. The group title links straight to the full detail page. + +Filters across the top: + +- **State** — All / Open / Resolved toggle. Defaults to **Open**. +- **Routing rule** — dropdown filtered to rules in the workspace. +- **Label matchers** — chip input where you type `key=value`. Press Enter to add a chip. + +Severity isn't a filter on the list — it lives on the per-instance signals inside a group, not on the group itself. + +## Anatomy of a group + +A group carries: + +- **Title** — set by the first signal that opened the group. **Immutable** after that, even if a higher-severity signal joins later. A lock icon next to the title indicates this; hovering it reads *"Set by first signal — cannot be changed."* +- **State** — `Open` while at least one underlying instance is firing; `Resolved` once everything quiets. A firing group whose firing instances are **all silenced** stays `Open` but carries a **Muted** badge (*"Firing, but all instances are silenced — notifications muted"*) — muting gates notifications, it doesn't resolve the group. +- **Labels** — the keys you grouped by on the routing rule, derived from the signals that joined the group. +- **Signal count** — total signals attached to the group so far. +- **Routing rule** — the rule that opened the group. +- **Group ID** — the durable identifier you can share with teammates or paste into the assistant. +- **Attached investigation** — present when [Auto-RCA](../auto-rca/) ran on the group. + +Severity isn't a single group-level field — each underlying instance carries its own severity, and you read them on the **Firing instances** panel and the **Signals** tab. + +## Group detail + +Click any row to open the group detail page. + +![Alert Group detail](./images/alert-group-detail.png) + +### Header + +- **Title** with a lock icon (hover: *"Set by first signal — cannot be changed"*). +- **Ask KloudMate Assistant** — opens the assistant chat panel with the group's labels, state, signal count, and routing rule pre-loaded into the prompt, so you can jump into investigation without retyping context. +- **Silence this group** — opens the silence creator pre-filled with the group's labels as matchers and bound to the group via `auto_expire_group_id`. + +### Meta strip + +Below the title: state chip, signal count (`N signals in this group`), Opened relative time, Resolved relative time (when applicable), and Last signal relative time. + +### Labels, routing rule, and Group ID + +A bordered card directly under the meta strip shows: + +- **Labels** — the group's label chips. Per-instance label keys (`alarm_id`, `instance_key`, etc.) are filtered out here so you see the labels that actually defined the group. +- **Routing rule** — a link to the rule that opened the group. +- **Group ID** — monospace, for sharing or pasting into tooling. + +### Firing instances + +The **Firing instances** panel is the dominant body section. It's a table of the unique alert instances that joined this group, deduplicated by the per-instance labels the grouping engine uses. + +- **Common labels bar** at the top — labels shared by every instance, so the per-row labels column only shows what varies. +- **Columns**: + - **State** — the instance's current state: **Firing**, **No Data**, **Error**, or **Resolved**. Silenced instances are flagged as muted, so you can see at a glance which ones are firing but suppressed. + - **Alarm** — the alert rule name. Links to the rule when the signal is KloudMate-native (carries an `alarm_id` label). + - **Instance labels** — the labels that distinguish this instance from siblings (common labels are stripped out). + - **Severity** — the per-instance severity, if the signal carries one. + - **Since** — how long this instance has been in its current state. +- A summary above the table reports the alert and instance counts with a per-state breakdown, e.g. *"2 alerts · 7 instances · 4 firing, 3 resolved"*. + +Use this panel to see what's firing inside the group at a glance — for example, service A's `prod` and `staging` are firing while `dev` has already recovered. + +### Attached investigation + +When the routing rule had Auto-RCA enabled, an info-styled card sits between the Firing instances panel and the tabs: + +- Investigation title and status chip (`Completed` / `In progress`). +- The root-cause analysis text. +- A **View full investigation** link. + +See [Auto-RCA](../auto-rca/) for how this gets created. + +### Tabs + +- **Signals** — every underlying firing as a row. Columns: Alert (linked to the rule when available), State, Severity, Received at, Labels. Use this when you need the raw event-by-event stream rather than the instance-level rollup. +- **Audit** — vertical timeline of everything that happened to the group: opened, appended, silence applied, notification dispatched, resolved. +- **Notifications** — per-channel dispatch outcomes (ok / failed / suppressed-by-silence), with deep links to where the notification landed: Slack thread URL, Jira ticket, KloudMate Incidents incident, etc. + +## When a group resolves + +KloudMate marks a group `Resolved` once every underlying signal has resolved and a short flap-grace period has elapsed (a few minutes — long enough to absorb signals that flap rapidly without closing the group prematurely). Resolved groups stay in the list under the **Resolved** filter and remain searchable indefinitely. + +## Related + +- [Routing Rules](../routing-rules/) — define which alerts a group accepts and how it's notified. +- [Silences](../silences/) — suppress notifications for matching labels. +- [Auto-RCA](../auto-rca/) — automatic investigations on group open. diff --git a/src/content/docs/docs/alerts/alert-lifecycle.mdx b/src/content/docs/docs/alerts/alert-lifecycle.mdx new file mode 100644 index 00000000..80b27c67 --- /dev/null +++ b/src/content/docs/docs/alerts/alert-lifecycle.mdx @@ -0,0 +1,136 @@ +--- +title: "Alert Lifecycle & States" +description: "How an alert moves through its states, and exactly how the pending duration, recovery period, and no-data / error settings shape that lifecycle." +sidebar: + order: 2.5 + label: "Lifecycle & States" +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +Once you've [created an alert](../create-alerts/), KloudMate evaluates it on a schedule and moves it through a series of states. The control knobs in **Configure evaluation settings** — **Evaluate every**, **Pending duration**, **Recovery period**, **Alert state if No data**, and **Alert state if Error** — each shape *when* and *how* those transitions happen. + +This page explains the lifecycle so the behavior never surprises you. If you've ever wondered "why did my alert fire instantly?" or "why didn't No Data wait out my pending duration?", the answer is here. + +## How evaluation works + +1. KloudMate runs the rule's queries and expressions on the cadence set by **Evaluate every** (e.g. every `60s`). +2. A single rule can produce many **instances** — one per series returned by the query (one per host, function, endpoint, …). Each instance is tracked independently and has its own state. +3. Every evaluation produces one raw **outcome** per instance, *before* any of the timing knobs are applied: + +| Outcome | What it means | +|---|---| +| **Breaching** | The alert condition evaluated to true. For a [Condition Expression](../expressions/), that's your threshold being crossed. | +| **Normal** | The alert condition evaluated to false. | +| **No Data** | The query returned no data points for the window (or the value was null / `NaN`). | +| **Error** | A query failed, or the rule couldn't be evaluated at all — bad query, source unreachable, or an evaluation timeout. | + +:::tip +A raw metric is treated as **breaching** whenever its value is non-zero. To compare against a threshold instead, wrap it in a **Condition Expression** so the outcome reflects `value > threshold` (true/false), not just "non-zero". See [Writing Expressions](../expressions/). +::: + +These raw outcomes are then fed through the timing knobs to produce the instance's **lifecycle state**. + +## Lifecycle states + +| State | Meaning | +|---|---| +| **Normal** | The condition is within threshold. Nothing to do. | +| **Pending** | The condition is currently breaching but hasn't held long enough to fire yet. | +| **Firing** | The condition has breached for at least the **Pending duration**. This is what notifies. +| **Recovering** | The condition has cleared, but the instance is still firing while it waits out the **Recovery period**. It has *not* resolved yet. | +| **No Data** | The query returned no data — and the rule is configured to surface that as its own state. | +| **Error** | The evaluation errored — and the rule is configured to surface that as its own state. | + +**Pending** and **Recovering** are *derived* states — KloudMate computes them from the raw outcome plus the timing knobs. The evaluator itself never reports them. + +## The firing episode + +The core of the lifecycle is the **firing episode** — the path an instance takes from quiet, to firing, and back to quiet: + +``` +Normal ──breach──▶ Pending ──held ≥ pending duration──▶ Firing ──clears──▶ Recovering ──clear held ≥ recovery period──▶ Normal +``` + +- **Pending duration** gates the way *in*. The condition must breach **continuously** for this long before the instance fires. A single Normal evaluation in the middle resets it back to Normal — the breach has to be sustained, not intermittent. Set it to `0` (leave it empty) to fire on the first breaching evaluation. +- **Recovery period** gates the way *out*. After an instance clears, it stays **Firing** (shown as **Recovering**) until the condition has stayed clear for this long. This stops a flapping metric from resolving and immediately re-firing — which, because a resolved alert is terminal, would otherwise open a brand-new alert group on every flap. Set it to `0` (leave it empty) to resolve the instant the condition clears. + +Exact transitions: + +| From | Evaluation | Goes to | When | +|---|---|---|---| +| Normal | breaching | **Pending** | Pending duration > 0 | +| Normal | breaching | **Firing** | Pending duration = 0 (fire immediately) | +| Pending | breaching | **Firing** | breach has been sustained ≥ Pending duration | +| Pending | breaching | **Pending** | still inside the Pending duration | +| Pending | clears | **Normal** | resets — the breach was never confirmed | +| Firing | clears | **Recovering** | Recovery period > 0 | +| Firing | clears | **Normal** (resolved) | Recovery period = 0 (resolve immediately) | +| Recovering | clears | **Normal** (resolved) | clear has been sustained ≥ Recovery period | +| Recovering | clears | **Recovering** | still inside the Recovery period | +| Recovering | breaching | **Firing** | a re-breach during recovery — snaps straight back to Firing | + +:::note +A **re-breach during Recovering** snaps the instance back to **Firing** without starting a fresh Pending wait and without re-notifying — from the episode's point of view it never stopped firing. Only a clear that survives the whole Recovery period actually resolves the alert. +::: + +## The control knobs at a glance + +| Knob (form label) | What it controls | Default | Inherits from [folder](../folders/)? | +|---|---|---|---| +| **Evaluate every** | How often the rule runs. | `60s` | Yes | +| **Pending duration** | How long a breach must hold before **Firing**. | Fire immediately | No — per rule | +| **Recovery period** | How long a clear must hold before resolving. | Resolve immediately | No — per rule | +| **Alert state if No data** | Which state a no-data result becomes. | No Data | Yes | +| **Alert state if Error** | Which state an error result becomes. | Error | Yes | + +## How No Data and Error fit in + +This is the part that most often surprises people, and it's the answer to a common question: + +> *I configured the same query, pending duration, and recovery period. Only the **Firing** state goes through the pending and recovery windows — **No Data** and **Error** don't. Is that expected?* + +**Yes — that's expected.** The **Pending duration** and **Recovery period** are properties of the *firing episode*. They gate the ramp **into** and **out of** Firing. **No Data** and **Error** are separate evaluation outcomes — by default they are not part of a firing episode, so there is nothing for those windows to debounce. An instance switches into (and out of) No Data or Error **immediately**. + +What each setting does on a no-data or error result: + +**Alert state if No data** + +| Target | Behavior when the query returns no data | +|---|---| +| **No Data** (default) | Instance switches to **No Data** immediately. No pending wait, no recovery wait. | +| **Normal** | Instance switches to **Normal** immediately — treated as a clear. | +| **Error** | Instance switches to **Error** immediately. | +| **Firing** | Treated exactly like a threshold breach: it ramps through the **Pending duration** on the way in, and once firing it honors the **Recovery period** when data returns. An already-firing instance rides through the gap without resetting. | + +**Alert state if Error** + +| Target | Behavior when the evaluation errors | +|---|---| +| **Error** (default) | Instance switches to **Error** immediately. No pending wait, no recovery wait. | +| **Normal** | Instance switches to **Normal** immediately — treated as a clear (so a firing instance enters the Recovery period). | +| **Firing** | Treated exactly like a threshold breach: it ramps through the **Pending duration**, and honors the **Recovery period** when the query recovers. | + +:::note[The takeaway] +If you want a no-data or error condition to be **debounced by the same Pending duration and Recovery period as a threshold breach**, set **Alert state if No data** / **Alert state if Error** to **Firing**. Left on their default (**No Data** / **Error**), they are immediate, deliberate signals — a gap or a broken query shows up right away rather than waiting out a timer. +::: + +:::tip +Infrastructure failures — a query timeout, an unreachable source, or any error that prevents the rule from running — are routed through **Alert state if Error**, exactly like an error *returned* by the query. So "couldn't evaluate" behaves consistently with how you configured the error state. +::: + +## Other behaviors you might notice + +- **Transient blips on a firing alert.** A firing instance only *rides through* a momentary No Data or Error gap when that state is mapped to **Firing**. If No Data / Error are left on their defaults, a firing instance switches to **No Data** / **Error** on the next bad evaluation — it does not stay "Firing" through the gap. +- **A series that disappears entirely.** When a whole series stops appearing in the query results (a host is decommissioned, a metric is dropped), KloudMate waits a short grace window — about **two evaluation intervals** — and then treats the vanished instance as resolved and closes it. This is distinct from **No Data**, which is the query *running* and returning nothing. +- **Multi-instance rules.** A rule's overall state reflects its instances: if any instance is firing (or recovering), the rule reads as **Firing**; otherwise it surfaces the most significant instance state. Each instance still runs its own lifecycle independently. +- **Silences and maintenance windows don't change state.** They gate the outbound *notification*, not the lifecycle. A silenced instance keeps evaluating and transitioning, and its state changes still land in history — only the notification is withheld. See [Silences](../silences/) and [Maintenance Windows](../maintenance-windows/). + +## Related + + + + + + + diff --git a/src/content/docs/docs/alerts/annotations-and-severity.mdx b/src/content/docs/docs/alerts/annotations-and-severity.mdx new file mode 100644 index 00000000..f0c92ad2 --- /dev/null +++ b/src/content/docs/docs/alerts/annotations-and-severity.mdx @@ -0,0 +1,136 @@ +--- +title: "Annotations & Severity" +description: "Attach Liquid-templated annotations to alerts and use the reserved severity key to drive downstream routing." +sidebar: + label: "Annotations & Severity" + order: 3 +--- + +import { Steps } from '@astrojs/starlight/components'; + +KloudMate attaches custom metadata to your alerts using **annotations** — key-value pairs that flow into notifications and incidents along with the alert itself. + +Annotation values support **Liquid templating**, so you can pull live data from the firing alert into the notification text. The reserved key `severity` drives how incident-management and routing tools prioritize the alert. + +## Where to find annotations + +Annotations live in the **Responder context** section of the alert rule editor (inside the **Add alarm details** step). Five annotation keys are pre-defined and render as labelled form fields: + +- **Severity** — free-form severity string, e.g. `sev1`, `critical`, `p1`. Plain text or Liquid template. +- **Summary** — multiline text, shown as the alert headline in notifications. Supports templates. +- **Dashboard** — pick a workspace dashboard to link from the notification. +- **Panel** — when a dashboard is picked, narrows the link to a specific panel. +- **Playbook URL** — link to the runbook responders should follow. Supports templates. + +Any other key is a **custom annotation**. Custom annotations live under a collapsible **Custom annotations** accordion below the fixed fields — click **Add Annotation** to create a new key-value row. + +## Add a custom annotation + + +1. **Open the rule editor.** Scroll to **Responder context** → expand the **Custom annotations** accordion. + +2. **Click Add Annotation.** A new key-value row appears. + +3. **Provide a key.** Use any string — e.g. `service_owner`, `region`, `runbook_section`. + +4. **Provide a value.** This can be static text (`payments-team`) or a Liquid template (`{{ labels.service }} is degraded`). + +5. **Save the rule.** Annotation values render at notification time, not at save time — see [Failure mode](#failure-mode) below. + + +## Using Liquid templates + +You can interpolate live alert data into any annotation value with the `{{ }}` syntax. The render context shape is: + +```jsonc +{ + "labels": { + "service": "checkout-api", + "env": "prod" + // ...plus any user-defined label keys from query dimensions or folders + }, + "state": { + "current_state": "Alerting", + "resolved": false, + "started_at": "2026-05-19T12:00:00Z", + "value": 0.05, // the evaluated value of the condition node + "values": { "A": 1000, "B": 50 } // raw outputs of each query/expression node + }, + "rule": { + "id": "alm_abc123", + "name": "High error rate", + "description": "Error rate ratio vs request volume" + } +} +``` + +### Common references + +- `{{ labels.service }}` — the value of the `service` label on the firing instance. +- `{{ state.value }}` — the evaluated number that crossed the threshold. +- `{{ state.values.A }}`, `{{ state.values.B }}` — raw outputs of each query / expression node (by node ID). +- `{{ state.current_state }}` — `Alerting`, `Normal`, etc. +- `{{ state.resolved }}` — boolean; useful in `{% if state.resolved %}…{% endif %}` branches. +- `{{ rule.name }}`, `{{ rule.id }}` — the alert rule's name and id. Prefer these over `labels.alarm_id` / `labels.alarm_name` — those internal keys are stripped before render. + +### Example: contextual summary + +If you watch CPU utilization with query `A` grouped by `host_name`, you might write: + +- **Key:** `Description` +- **Value:** `Host {{ labels.host_name }} is at {{ state.value }}% CPU.` + +When the alert fires for `web-server-01` at 95%, the notification reads: *Host web-server-01 is at 95% CPU.* + +## Testing a template + +Every templated value field has a **Test template** icon to its right. Click it to open a side drawer: + +- The drawer pre-loads the Monaco editor with your current template. +- A **Sample context** panel lets you tweak the labels / state / rule JSON the template renders against. +- Click **Test** to see the rendered output below. Parse errors surface inline so you can fix them before saving. + +The drawer is purely opt-in — KloudMate doesn't validate templates on save. If a template can't render at notification time, KloudMate falls back to the raw template string rather than dropping the notification. + +### Failure mode + +- A broken template stays in the value field as-is when you save. +- At render time, a parse failure makes that single annotation surface the raw template text instead of an evaluated value. +- Other annotations on the same alert keep rendering normally. + +This fail-soft behavior is intentional: a typo shouldn't silence a real incident. + +## Dynamic severity + +Severity flows out of the reserved `severity` annotation. It's a string that downstream notification channels and incident-management tools use to prioritize the alert. Combine Liquid with `if` / `elsif` to compute severity from the firing value. + +### Example: severity by threshold + +Suppose query `B` is an error-rate ratio. You want: + +- Error rate > 10% → `critical` +- Error rate > 5% → `warning` +- Anything below that → `info` + +Set the `severity` annotation value to: + +```liquid +{% if state.value > 0.1 %}critical{% elsif state.value > 0.05 %}warning{% else %}info{% endif %} +``` + +At notification time, KloudMate evaluates the template against the firing alert's state and writes the resolved string into the outgoing payload. + +## Severity vs. tags + +Earlier versions of KloudMate used **notification tags** on the rule as the routing key. Tags have been replaced by: + +- **Labels** — auto-derived from query dimensions and folders. These are what [Routing Rules](../routing-rules/) match against. +- **Annotations** — human-readable context that flows into the notification. +- **Severity** — the reserved annotation key, free-form, optionally Liquid-templated. + +If you used to route by tag, switch to a routing rule that matches on the equivalent label (e.g. `service=checkout-api`). + +## Related + +- [Routing Rules](../routing-rules/) — how labels match alerts to channels. +- [Creating Alerts](../create-alerts/) — where annotations live in the rule editor. diff --git a/src/content/docs/docs/alerts/auto-rca.mdx b/src/content/docs/docs/alerts/auto-rca.mdx new file mode 100644 index 00000000..55a9e5a0 --- /dev/null +++ b/src/content/docs/docs/alerts/auto-rca.mdx @@ -0,0 +1,46 @@ +--- +title: "Auto-RCA" +description: "Automatically launch an AI investigation when an alert group opens." +sidebar: + order: 8 +--- + +Auto-RCA hands an [Alert Group](../alert-groups/) to KloudMate's AI investigator the moment it opens, so by the time you open the group there's already a root-cause summary waiting. You enable it per [Routing Rule](../routing-rules/) — every group the rule produces gets investigated. + +## How it works + +When a routing rule with Auto-RCA enabled opens a group, KloudMate schedules an investigation after the configurable **Auto-RCA delay** (default: 5 minutes). The delay gives related signals time to fold into the group so the investigation sees the full picture, not only the first signal. + +When the investigation completes, its summary attaches to the group: + +- The group detail page shows the **Attached investigation** block with the summary excerpt and a **View full investigation** link. +- The notifications dispatched from the group include the investigation summary (where the channel format supports it — Slack thread reply, KloudMate Incidents comment, etc.). + +:::note[One investigation per group] +Auto-RCA runs only once per alert group — when the group is first opened. Alerts that later join an existing group (for example, a Memory alert folding into a group that a CPU alert already opened) do **not** trigger a new investigation. Those alerts still send the usual group-transition notifications, but no additional Auto-RCA report is generated for them. +::: + +:::note[Placeholder image] +Screenshot pending — an alert group detail page showing an attached investigation block with summary excerpt and the View full investigation link. +::: +![Attached investigation on group detail](./images/alert-group-investigation.png) + +## Enable Auto-RCA on a routing rule + +1. Open **Alerts → Routing Rules** and edit the rule you want to enrich. +2. Scroll to the **Auto-RCA** section. +3. Toggle **Auto-RCA** on. +4. Set the **Delay** if the default 5 minutes isn't right. A shorter delay surfaces answers faster; a longer one captures more signals before the investigation runs. +5. Save. + +Every group this rule opens from now on will trigger an investigation after the configured delay. + +## When Auto-RCA is most useful + +Auto-RCA works best on rules that bundle related signals — a spike after a deploy, cascading errors across services, or downstream impact from an upstream incident. For very narrow per-host or per-instance rules, the investigation has less to work with and you may prefer to disable it to reduce compute usage. + +## Related + +- [Routing Rules](../routing-rules/) — where the Auto-RCA toggle lives. +- [Alert Groups](../alert-groups/) — where the investigation surfaces. +- [KloudMate Assistant](../../kloudmate-assistant/) — the broader investigator surface. diff --git a/src/content/docs/docs/alerts/create-alerts.mdx b/src/content/docs/docs/alerts/create-alerts.mdx new file mode 100644 index 00000000..12a06fc2 --- /dev/null +++ b/src/content/docs/docs/alerts/create-alerts.mdx @@ -0,0 +1,198 @@ +--- +title: "Creating Alerts" +description: "Step-by-step guide to setting up and configuring KloudMate alerts." +sidebar: + order: 2 +--- +KloudMate lets you create and configure alerts for events that are critical to your application. Each alert watches a query, evaluates a condition, and — when the condition holds long enough — fires into the grouping and routing engine so the right people are notified. + +## Getting started + +Navigate to the **Alerts** section from the left navigation menu. + +![Alerts list](./images/setting-up-kloudmate-alarms-1.png) + +The **Alerts** screen displays a list of all existing alert rules along with their current state, name, and description. The summary at the top shows the total number of rules, including how many are currently **Firing** or **Pending**. Rules can be grouped into [Folders](../folders/), which appear as collapsible header rows. + +From the **more options (⋯)** icon on any rule, you can: + +- **View** the rule details +- **View State History** for the rule +- **Edit** the rule configuration +- **Duplicate** the rule +- **Pause Evaluation** or **Pause Notifications** +- **Delete** the rule + +![Alert row actions](./images/setting-up-kloudmate-alarms-1.jpeg) + +**Pause Notifications** opens a dialog that creates a temporary [silence](../silences/) scoped to this alert: the alert keeps evaluating, but its notifications are suppressed until the silence expires. While it's paused the rule shows a **Silenced** badge on the alerts list and on its detail page; view or end the pause from **Alerts → Silences**. Add matchers to limit it to specific instances. + +To learn about the key concepts of KloudMate Alerts, see the [Alerts Overview](../). + +## Creating a new alert + +Click the **Create Alert** button at the top-right corner of the **Alerts** screen. A dialog appears with three ways to create an alert: + +![Create Alert dialog](./images/create-alarm-dialog.png) + +- **From Template** — Start with a pre-configured alert for a common monitoring scenario. +- **From Scratch** — Create a custom alert from an empty configuration. +- **Using AI** — Describe the alert you want in plain English and let KloudMate build it for you. + +### From Template + +Instead of building an alert from scratch, you can start from a pre-configured template that covers common monitoring scenarios and best practices. + +1. In the **Create Alert** dialog, select **From Template**. +2. Click the **Select a template** dropdown and choose a template that matches your monitoring needs. + +![Template selection dropdown](./images/from-template-dropdown.png) + +3. Click **Create Alert**. The alert is created and appears in the **Alerts** list. +4. To open and configure it, click the menu next to the alert and select **Edit**. +5. The alert opens pre-configured with the query, aggregation, and threshold settings from the template. Review and adjust any filters to match your environment. +6. Click **Save** or **Save & Close** when done. + +### Using AI + +KloudMate's assistant can automatically generate queries and thresholds based on your natural language prompt. + +1. In the **Create Alert** dialog, select **Using AI**. +2. A text box appears. Describe the alert you want to create. +3. Click **Create Alert**. KloudMate generates the alert configuration based on your description. +4. To review or adjust the settings, click the menu next to the alert and select **Edit**. + +### From Scratch + +To build a fully custom alert, select **From Scratch** in the **Create Alert** dialog and click **Create Alert**. + +This opens the alert creation form where you can choose a data source, configure the metric or query to monitor, and define the alert condition on a single page. + +You can create multiple queries and expressions using the **Add Query** and **Add Expression** buttons. Each query or expression is assigned a unique alphabetical notation such as **A**, **B**, or **C**. You can duplicate any query or expression using the copy icon at the top-right corner of each block. + +To access advanced query and expression options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode** at the top of the form. + +![Advanced mode options](./images/advanced-mode-options.png) + +The rule editor walks you through four steps: **Setup query conditions and expressions**, **Configure evaluation settings**, **Add alarm details**, and **Notifications**. + +### 1. Setup query conditions and expressions + +#### Setting up query conditions for OpenTelemetry / KloudMate + +![OpenTelemetry / KloudMate query setup](./images/setting-up-kloudmate-alarms-2.png) + +- **Data Set:** Select the dataset you want to retrieve from your data source. +- **Metric to Aggregate:** Select the metric associated with the selected dataset that you want to monitor. +- **Group By:** Enter the attributes used to group the data points. +- **Filters:** Add filters to narrow down the retrieved data points. + +OpenTelemetry users can also use **Prometheus query language** to retrieve data and configure alerts. + +#### Setting up query conditions for AWS (CloudWatch) + +![AWS CloudWatch query setup](./images/setting-up-kloudmate-alarms-3.png) + +- **Time Range:** Set the duration for which data should be fetched using the dropdown, or enter a custom value in seconds. +- **Region:** Select the AWS region of the service you want to monitor. +- **Namespace:** Select the AWS service namespace you want to create an alert for. +- **Metric:** Select the metric associated with the selected namespace. +- **Statistic:** Select the statistical function to use when calculating data points. +- **Dimensions:** Optionally configure the alert for grouped resources within the selected namespace. For example, for EC2, you can filter by autoscaling group name, image ID, instance type, and more. + +Click **Run Query** to fetch data. + +#### Time range expressions for custom time + +Query time ranges support the following: + +- **Operators:** `-` for subtracting time +- **Supported values:** The same units and keywords used in dashboards +- **Examples:** `now`, `now-5m` + +#### Setting up evaluation expressions + +Expressions let you apply logic to query results. Reference any configured query or expression using its alphabetical notation, such as **A**, **B**, or **C**. An expression can be passed as a parameter only when multiple expressions are configured. + +Choose from the following expression types: + +- **Math Expression:** Enter a mathematical expression to apply to the value of a query or expression. Examples: `$A+1`, `$A<$B`, `$A && $C`. For more information, see [Alert Expressions](../expressions/). +- **Reduce:** Select a function to aggregate the values of a query or expression into a single number, then select the target query or expression from the **Input** dropdown. Available functions include `mean()`, `max()`, `min()`, `sum()`, `last()`, and `count()`. +- **Condition Expression:** Select a function and a query or expression, then choose a condition and provide a threshold value to evaluate against. You can add multiple conditions and combine them using **AND** or **OR** logical operators. + +Click **Run Queries** to execute all configured queries and expressions. + +:::tip +To avoid the **NoData** issue when using multiple queries in a single alert, use the `ifNull` operator to assign a default value. Read more in [Alert Expressions](../expressions/). +::: + +### 2. Configure evaluation settings + +This section opens with a **Folder** dropdown — select an existing folder (or type a new name to create one inline) to organize the rule and inherit shared defaults. The folder's `interval_seconds`, `no_data_state`, and `eval_error_state` flow into the rule as inheritable defaults you can override per-field. See [Folders](../folders/). + +![Evaluation settings](./images/setting-up-kloudmate-alarms-4.png) + +- **Alert condition** — Select the query or expression that should trigger the alert, such as **A**, **B**, or **C**. +- **Evaluate every** — How frequently the alert condition should be evaluated (e.g. `60s`, `1m`). +- **Pending duration** — How long the condition must remain true before the alert fires (e.g. `5m`). Leave empty to fire immediately. +- **Recovery period** — How long the condition must stay within threshold before the rule resolves (e.g. `5m`), which stops a flapping metric from resolving and immediately re-firing. While the rule waits out this window, the instance shows as **Recovering** — still firing, not yet back to Normal. Leave empty to resolve as soon as the condition clears. +- **Alert state if No data** — Which state the alert enters when the query returns no data points. Options: **Firing**, **No Data**, **Normal**, or **Error**. +- **Alert state if Error** — Which state the alert enters when a query returns an error. Options: **Firing**, **Error**, or **Normal**. + +:::note[How “Firing” interacts with the pending duration] +When **Alert state if No data** or **Alert state if Error** is set to **Firing**, the alert doesn't fire the instant data goes missing or a query errors. It fires only after the condition has held for the **Pending duration** above — so a brief gap or a transient error won't page anyone, and a series that's already firing rides through the blip without resetting. Set the **Pending duration** to `0` to fire immediately instead. +::: + +:::tip +For the full lifecycle — how **Pending duration**, **Recovery period**, and the **No data** / **Error** states interact across an alert's states — see [Alert Lifecycle & States](../alert-lifecycle/). +::: + +Click **Preview alerts** to run the query immediately and check the result. + +:::tip +If the rule belongs to a folder with shared defaults, **Evaluate every**, **Alert state if No data**, and **Alert state if Error** appear with placeholder values inherited from the folder. Leave them blank to keep inheriting, or type a value to override. A small **Reset to folder default** link appears under any field you've overridden. +::: + +### 3. Add alarm details + +![Alert details](./images/setting-up-kloudmate-alarms-5.png) + +Two top-level fields and one section: + +- **Alert name** — Enter a name for the alert. +- **Description** — Add a description to help identify the alert's purpose. + +**Responder context** — a labelled section that holds the five fixed annotations responders see when the notification lands. The hint above the section reads e.g. *"Help on-call responders understand the alarm and act quickly."* + +- **Severity** — Free-form severity (e.g. `sev1`, `critical`, `p1`). Supports Liquid templates so severity can depend on the firing value. +- **Summary** — Multiline message included in notifications. Supports templates. +- **Dashboard** — Optional dashboard link surfaced with the notification. +- **Panel** — When a dashboard is picked, narrows the link to a specific panel. +- **Playbook URL** — Optional runbook URL. + +**Custom annotations** — collapsed accordion at the bottom. Open it to add custom key-value pairs (e.g. `service_owner`, `region`). Values support Liquid templates. See [Annotations & Severity](../annotations-and-severity/). + +### 4. Notifications + +The Notifications step covers how this alert flows into the grouping engine. KloudMate replaced free-form notification tags with **labels** that routing rules match on: + +- **Labels** are auto-derived from query dimensions, the alert rule's folder, and a small set of system keys. You don't enter them by hand. +- **Routing** is decided by [Routing Rules](../routing-rules/) in **Alerts → Routing rules**, which match alerts by label and send them to one or more notification channels. +- **Severity** flows through the reserved `severity` annotation from the previous step — downstream tools use it to prioritize. + +If you're migrating an existing rule that used notification tags, the rule keeps working; routing now matches on labels instead. + +### 5. Save the alert + +Click **Save** to save the alert, or **Save & Close** to save and return to the **Alerts** screen. A confirmation message appears when the alert is created successfully. + +## Viewing an alert + +To open an alert, click the menu next to it and select **View**. This opens the detail page with four tabs: + +![Alert detail overview](./images/alarm-detail-overview.png) + +- **Overview** — Shows instance states, breaching instances with labels, reason, and duration, along with recent state transitions. +- **Instances** — Shows the full list of alert instances and their current states. +- **History** — Shows the state change history over time. +- **Rule** — Shows the alert configuration and query definition. diff --git a/src/content/docs/docs/alarms/expressions.mdx b/src/content/docs/docs/alerts/expressions.mdx similarity index 87% rename from src/content/docs/docs/alarms/expressions.mdx rename to src/content/docs/docs/alerts/expressions.mdx index 57ccc680..ace30c9f 100644 --- a/src/content/docs/docs/alarms/expressions.mdx +++ b/src/content/docs/docs/alerts/expressions.mdx @@ -1,16 +1,16 @@ --- -title: "Alarm Expressions" -description: "Learn how to write and use expressions to evaluate metrics in KloudMate Alarms." +title: "Writing Expressions" +description: "Write and use expressions to evaluate metrics in KloudMate alerts." sidebar: - order: 3 + order: 10 --- -Alarm expressions allow you to manipulate query results using math and other operations. KloudMate alarms can only target numeric data. If a query returns time series data, it must be reduced using expressions before it can be used as an alarm target. +Expressions let you manipulate query results using math and other operations. Alerts can only target numeric data, so if a query returns a time series, you must reduce it with an expression before it can drive an alert. Each query or expression is represented by a unique alphabetical notation (A, B, C, and so on). While writing an expression, one or more queries or other expressions can be passed as parameters using their notation. -## Expression Operations +## Expression operations -KloudMate supports three types of operations in alarm expressions: +KloudMate supports three types of operations in alert expressions: - Math expression - Reduce - Condition expression @@ -64,7 +64,7 @@ For the above operations: #### Using Expressions for Multiple Query Conditions -When working with multiple queries in a single alarm: +When working with multiple queries in a single alert: - Use the Reduce function for all time series outputs (e.g., mean, max, last). - Use the `ifNull` logic with all reduced values so that if any query returns NoData, it does not affect the evaluation of other queries. @@ -128,15 +128,15 @@ You can add multiple conditions in a condition expression using the **Add condit ## Nodes -Nodes represent individual queries or expressions used in the creation of alarms and dashboards. Each node is assigned an alphabetical identifier by default (e.g., A, B, C), based on the order in which it was created. +Nodes represent individual queries or expressions used in the creation of alerts and dashboards. Each node is assigned an alphabetical identifier by default (e.g., A, B, C), based on the order in which it was created. ![image](./images/writing-expressions-for-kloudmate-alarms-1.jpeg) -Nodes serve as a reference point, allowing users to easily identify and evaluate specific queries or expressions during data analysis. +Nodes serve as reference points, allowing users to identify and evaluate specific queries or expressions during analysis. ![image](./images/writing-expressions-for-kloudmate-alarms-2.jpeg) -While the default alphabetical naming is applied automatically, users have the flexibility to rename nodes for better clarity and easier identification. +While default alphabetical naming is applied automatically, nodes can be renamed for better clarity. ![image](./images/writing-expressions-for-kloudmate-alarms-3.jpeg) diff --git a/src/content/docs/docs/alerts/folders.mdx b/src/content/docs/docs/alerts/folders.mdx new file mode 100644 index 00000000..ef042408 --- /dev/null +++ b/src/content/docs/docs/alerts/folders.mdx @@ -0,0 +1,81 @@ +--- +title: "Folders" +description: "Group alert rules into folders and share evaluation defaults across the members." +sidebar: + order: 4 +--- + +Folders organize related alert rules and let member rules inherit shared evaluation defaults. They follow the dashboards-style grouping pattern applied to alerts: opt into a folder view, see your rules collapsed under their owning folder, and let one set of defaults propagate to whichever rules need them. + +On the **Alerts** list, the layout toggle in the toolbar switches between a flat **List view** and a grouped **Folder view**. In folder view, each folder renders as a collapsible header row with its member rules underneath; rules without a folder land in a pinned **Ungrouped** section. + +![Alerts list in folder view](./images/alerts-folder-view.png) + +There's no separate Folders page or sidebar entry — everything happens from the Alerts list. + +## Create a folder + +Folders are created inline from the alert rule editor's **Folder** picker — there's no separate "New folder" button. + +1. Open an alert rule (create a new one or edit an existing one). +2. In the **Configure evaluation settings** section, click the **Folder** dropdown. +3. Type a new folder name. A `+ Create ""` option appears at the bottom of the dropdown — click it. +4. KloudMate creates the folder and assigns the current rule to it. A toast confirms the creation. +5. To set folder-level defaults (interval, no-data state, eval-error state), switch to **Folder view** on the Alerts list, open the folder's kebab menu, and select **Edit folder**. + +## Edit folder defaults + +In Folder view, each folder header row has a kebab menu with **Edit folder** and **Delete folder**. + +**Edit folder** opens a dialog with the folder's settings: + +| Field | What it controls | +|---|---| +| **Name** | Required, unique within the workspace. | +| **Description** | Optional. | +| **Evaluate every** | Default evaluation interval (e.g. `60s`, `1m`, `2h`). Members inherit when blank. | +| **No data state** | Default state when queries return no data. Options: `Normal`, `Alerting`, `NoData`. | +| **Eval error state** | Default state when evaluation errors out. Options: `Normal`, `Alerting`, `Error`. | + +A note above these reads: *"Member alerts inherit these defaults unless they set their own value. Leave blank for organizational-only folders."* + +## Move rules into a folder + +Two ways: + +- **From the rule editor** — open the rule and pick the folder from the **Folder** dropdown in the **Configure evaluation settings** section. +- **Bulk-move from the list** — select rules using the checkboxes (a toolbar appears with the selection count). Click **Move to folder…**, select a destination from the radio list (existing folders or **Ungrouped**), and confirm. + +Moving a rule does **not** rewrite its `interval_seconds`, `no_data_state`, or `eval_error_state` columns. Whatever the rule had before the move stays. The folder defaults only apply where the rule's field is blank. + +## Inheritance and overrides + +A rule's eval-config fields follow a clear three-state model: + +| Rule field value | What the editor shows | What runs | +|---|---|---| +| Blank (inheriting) | Folder default as placeholder + `Inheriting from folder ` helper line | The folder default | +| Filled (overriding) | The rule's stored value + `Overrides folder default` helper line + **Reset to folder default** link | The rule's value | +| No folder selected | Plain input, no helper text | The rule's value (or system default if blank) | + +Clicking **Reset to folder default** clears the field and flips the helper text back to inheriting. Saving with the field blank persists `NULL` for that column — so when the folder default later changes, the rule picks up the new value automatically. + +:::tip +To detach a rule from its folder entirely, change the **Folder** dropdown to the ungrouped option. The eval-config columns keep their stored values; the rule stops inheriting. +::: + +## Delete a folder + +From the folder kebab menu, pick **Delete folder**. The confirmation explains: e.g. *"Member alerts will become ungrouped — their stored eval-config values stay intact."* KloudMate detaches the members first and then deletes the folder row. Rules survive the deletion. + +## Folder labels in alert groups + +A rule's folder identity flows into the alert's labels as `alarm_rule_folder_id` and `alarm_rule_folder_name`. That means: + +- [Routing Rules](../routing-rules/) can match on folder identity (e.g. `alarm_rule_folder_name in [Kubernetes, Networking]`). +- The folder name shows as a label chip on the [Alert Group](../alert-groups/) detail header automatically. + +## Related + +- [Creating Alerts](../create-alerts/) — picking a folder during rule creation. +- [Routing Rules](../routing-rules/) — matching on folder labels. diff --git a/src/content/docs/docs/alarms/images/advanced-mode-options.png b/src/content/docs/docs/alerts/images/advanced-mode-options.png similarity index 100% rename from src/content/docs/docs/alarms/images/advanced-mode-options.png rename to src/content/docs/docs/alerts/images/advanced-mode-options.png diff --git a/src/content/docs/docs/alarms/images/alarm-detail-overview.png b/src/content/docs/docs/alerts/images/alarm-detail-overview.png similarity index 100% rename from src/content/docs/docs/alarms/images/alarm-detail-overview.png rename to src/content/docs/docs/alerts/images/alarm-detail-overview.png diff --git a/src/content/docs/docs/alarms/images/alarm-maintenance-window-1.jpeg b/src/content/docs/docs/alerts/images/alarm-maintenance-window-1.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/alarm-maintenance-window-1.jpeg rename to src/content/docs/docs/alerts/images/alarm-maintenance-window-1.jpeg diff --git a/src/content/docs/docs/alarms/images/alarm-maintenance-window-1.png b/src/content/docs/docs/alerts/images/alarm-maintenance-window-1.png similarity index 100% rename from src/content/docs/docs/alarms/images/alarm-maintenance-window-1.png rename to src/content/docs/docs/alerts/images/alarm-maintenance-window-1.png diff --git a/src/content/docs/docs/alarms/images/alarm-maintenance-window-2.jpeg b/src/content/docs/docs/alerts/images/alarm-maintenance-window-2.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/alarm-maintenance-window-2.jpeg rename to src/content/docs/docs/alerts/images/alarm-maintenance-window-2.jpeg diff --git a/src/content/docs/docs/alarms/images/alarm-maintenance-window-3.jpeg b/src/content/docs/docs/alerts/images/alarm-maintenance-window-3.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/alarm-maintenance-window-3.jpeg rename to src/content/docs/docs/alerts/images/alarm-maintenance-window-3.jpeg diff --git a/src/content/docs/docs/alerts/images/alert-group-detail.png b/src/content/docs/docs/alerts/images/alert-group-detail.png new file mode 100644 index 00000000..8481f096 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-group-detail.png differ diff --git a/src/content/docs/docs/alerts/images/alert-group-investigation.png b/src/content/docs/docs/alerts/images/alert-group-investigation.png new file mode 100644 index 00000000..096476a7 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-group-investigation.png differ diff --git a/src/content/docs/docs/alerts/images/alert-groups-list.png b/src/content/docs/docs/alerts/images/alert-groups-list.png new file mode 100644 index 00000000..c2e5ae28 Binary files /dev/null and b/src/content/docs/docs/alerts/images/alert-groups-list.png differ diff --git a/src/content/docs/docs/alerts/images/alerts-folder-view.png b/src/content/docs/docs/alerts/images/alerts-folder-view.png new file mode 100644 index 00000000..06f85d3b Binary files /dev/null and b/src/content/docs/docs/alerts/images/alerts-folder-view.png differ diff --git a/src/content/docs/docs/alarms/images/create-alarm-dialog.png b/src/content/docs/docs/alerts/images/create-alarm-dialog.png similarity index 100% rename from src/content/docs/docs/alarms/images/create-alarm-dialog.png rename to src/content/docs/docs/alerts/images/create-alarm-dialog.png diff --git a/src/content/docs/docs/alarms/images/from-template-dropdown.png b/src/content/docs/docs/alerts/images/from-template-dropdown.png similarity index 100% rename from src/content/docs/docs/alarms/images/from-template-dropdown.png rename to src/content/docs/docs/alerts/images/from-template-dropdown.png diff --git a/src/content/docs/docs/alerts/images/routing-rules-list.png b/src/content/docs/docs/alerts/images/routing-rules-list.png new file mode 100644 index 00000000..5acb219d Binary files /dev/null and b/src/content/docs/docs/alerts/images/routing-rules-list.png differ diff --git a/src/content/docs/docs/alerts/images/routing-rules-suggest.png b/src/content/docs/docs/alerts/images/routing-rules-suggest.png new file mode 100644 index 00000000..0df4448a Binary files /dev/null and b/src/content/docs/docs/alerts/images/routing-rules-suggest.png differ diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-1.jpeg b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-1.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-1.jpeg rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-1.jpeg diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-1.png b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-1.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-1.png rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-1.png diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-2.png b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-2.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-2.png rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-2.png diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-3.png b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-3.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-3.png rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-3.png diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-4.png b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-4.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-4.png rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-4.png diff --git a/src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-5.png b/src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-5.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-kloudmate-alarms-5.png rename to src/content/docs/docs/alerts/images/setting-up-kloudmate-alarms-5.png diff --git a/src/content/docs/docs/alarms/images/setting-up-notifications-2.png b/src/content/docs/docs/alerts/images/setting-up-notifications-2.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-notifications-2.png rename to src/content/docs/docs/alerts/images/setting-up-notifications-2.png diff --git a/src/content/docs/docs/alarms/images/setting-up-notifications-3.jpeg b/src/content/docs/docs/alerts/images/setting-up-notifications-3.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-notifications-3.jpeg rename to src/content/docs/docs/alerts/images/setting-up-notifications-3.jpeg diff --git a/src/content/docs/docs/alarms/images/setting-up-notifications-3.png b/src/content/docs/docs/alerts/images/setting-up-notifications-3.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-notifications-3.png rename to src/content/docs/docs/alerts/images/setting-up-notifications-3.png diff --git a/src/content/docs/docs/alarms/images/setting-up-notifications-4.png b/src/content/docs/docs/alerts/images/setting-up-notifications-4.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-notifications-4.png rename to src/content/docs/docs/alerts/images/setting-up-notifications-4.png diff --git a/src/content/docs/docs/alarms/images/setting-up-notifications-5.png b/src/content/docs/docs/alerts/images/setting-up-notifications-5.png similarity index 100% rename from src/content/docs/docs/alarms/images/setting-up-notifications-5.png rename to src/content/docs/docs/alerts/images/setting-up-notifications-5.png diff --git a/src/content/docs/docs/alerts/images/silences-list.png b/src/content/docs/docs/alerts/images/silences-list.png new file mode 100644 index 00000000..de1b533c Binary files /dev/null and b/src/content/docs/docs/alerts/images/silences-list.png differ diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-1.jpeg b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-1.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-1.jpeg rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-1.jpeg diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-1.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-1.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-1.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-1.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-2.jpeg b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-2.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-2.jpeg rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-2.jpeg diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-2.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-2.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-2.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-2.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-3.jpeg b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-3.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-3.jpeg rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-3.jpeg diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-3.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-3.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-3.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-3.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-4.jpeg b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-4.jpeg similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-4.jpeg rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-4.jpeg diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-4.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-4.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-4.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-4.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-5.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-5.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-5.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-5.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-6.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-6.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-6.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-6.png diff --git a/src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-7.png b/src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-7.png similarity index 100% rename from src/content/docs/docs/alarms/images/writing-expressions-for-kloudmate-alarms-7.png rename to src/content/docs/docs/alerts/images/writing-expressions-for-kloudmate-alarms-7.png diff --git a/src/content/docs/docs/alerts/index.mdx b/src/content/docs/docs/alerts/index.mdx new file mode 100644 index 00000000..2c7e5c78 --- /dev/null +++ b/src/content/docs/docs/alerts/index.mdx @@ -0,0 +1,93 @@ +--- +title: "Alerts Overview" +description: "Understand how KloudMate alert rules, groups, routing rules, silences, folders, and annotations work together." +sidebar: + order: 1 + label: "Introduction" +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +Alerts tell you when something needs attention — an error-rate spike, a service going quiet, a budget threshold crossed. You write the rule once; KloudMate evaluates it on a schedule, groups related firings together, and routes notifications to the channels you care about. + +This page covers the concepts behind the Alerts module. Once they click, the rest of the section is a tour of each surface. + +## Key concepts + +### Alert rule + +An alert rule contains the evaluation criteria — one or more queries, expressions, and a condition. It also specifies how often KloudMate evaluates the rule, how long the condition must hold before the rule fires, and how long it must stay clear before the rule resolves. + +### Alert query + +A query fetches data from a source. KloudMate alerts support AWS CloudWatch and KloudMate's own telemetry (logs, metrics, spans) as sources. + +### Alert expressions + +Expressions transform query output with math, reductions, or conditions. They reduce time series to a single number you can compare against a threshold. See [Alert Expressions](./expressions/). + +### Alert labels + +Labels are key-value pairs attached to each firing alert. They come from query dimensions and any folder the rule lives in, and they're what [Routing Rules](./routing-rules/) match against to decide where notifications go. KloudMate generates labels automatically — you don't enter them by hand. + +### Annotations and severity + +Annotations carry human-readable context (summary, runbook URL, dashboard link) into each notification. Values support Liquid templates, so you can interpolate live label and state data into the message. The reserved `severity` annotation key drives downstream routing and incident severity. See [Annotations & Severity](./annotations-and-severity/). + +### Folders + +Folders group related alert rules and let member rules inherit shared defaults — evaluation interval, no-data state, and eval-error state. See [Folders](./folders/). + +### Alert groups + +When matching alerts fire close together, KloudMate bundles them into a single **Alert Group** — a durable, deduplicated container that updates as higher-severity signals join, holds the auto-RCA if you've enabled one, and provides a single thread to notify against. See [Alert Groups](./alert-groups/). + +### Routing rules + +Routing rules decide which channels get notified for which alerts. Each rule matches alerts by their labels, groups them by chosen label keys, and dispatches notifications to one or more channels. See [Routing Rules](./routing-rules/). + +### Silences + +A silence suppresses notifications for matching labels for a bounded time window — useful when you know an alert will be noisy and don't want to flood your channels. See [Silences](./silences/). + +### Multi-dimensional alerts + +A single rule can produce multiple alert instances, one per dimension. A rule watching Lambda throttling generates one instance per throttled function. + +## Workflow of KloudMate Alerts + +1. A rule retrieves data from its source using queries. +2. Expressions reduce or transform query results. +3. The condition is evaluated; if it holds for the pending duration, the rule fires. +4. The firing alert is tagged with labels and annotations and sent into the grouping engine. +5. The grouping engine matches the alert against [Routing Rules](./routing-rules/), opens or appends to an [Alert Group](./alert-groups/), and dispatches notifications to the rule's destination channels. +6. Active [Silences](./silences/) and [Maintenance Windows](./maintenance-windows/) can suppress the outbound notification at this step. The rule keeps evaluating in the background and the state change still lands in history. + +## KloudMate alert states + +- **Firing / Alerting** — the rule's condition has held longer than the pending duration. +- **Pending** — the condition is currently true but hasn't held long enough yet. +- **Recovering** — the condition has cleared, but the rule is still firing while it waits out the recovery period before resolving. +- **Normal** — everything's quiet. +- **Error** — the evaluation hit an error (bad query, source unreachable). +- **No Data** — the query returned no data for the configured window. + +For how these states transition — and how the pending duration, recovery period, and no-data / error settings shape them — see [Alert Lifecycle & States](./alert-lifecycle/). + +## Next steps + + + + + + + + + +## Related resources + + + + + + diff --git a/src/content/docs/docs/alerts/maintenance-windows.mdx b/src/content/docs/docs/alerts/maintenance-windows.mdx new file mode 100644 index 00000000..41dd34ff --- /dev/null +++ b/src/content/docs/docs/alerts/maintenance-windows.mdx @@ -0,0 +1,56 @@ +--- +title: "Maintenance Windows" +description: "Schedule recurring or one-time maintenance windows that suppress alert notifications during planned downtime." +sidebar: + order: 9 +--- +A Maintenance Window defines a scheduled period during which matching alerts continue to evaluate but their notifications are suppressed. State changes are still recorded in alert history (marked with `silenced_at` / `silenced_by_*` so you can audit what happened), but no notification fires. Like a [Silence](../silences/), a maintenance window is a **notification gate**: it mutes notifications for the matching alerts but never changes their state or resolves them. + +Use a maintenance window when you have a calendar-driven quiet period: a planned deployment, a recurring batch job window, a vendor outage you've been told to expect. For one-off, ad-hoc suppression in the moment, use [Silences](../silences/) instead. + +## Purpose + +- Suppresses notifications for expected alerts during deployments, patches, upgrades, or infrastructure changes. +- Avoids unnecessary escalations or incident tickets. +- Keeps monitoring reports clean and meaningful. + +## Overview page + +The overview page displays all created maintenance windows with these details: + +- **Name** — Name of the maintenance window. +- **Description** — Description of the maintenance window. +- **Alert rule targeted** — Alert rule to which the maintenance window applies. +- **Matching label** — Labels assigned to the maintenance window. +- **Schedule** — Start and end time period for the maintenance window. +- **Status** — Whether the maintenance window is active or paused. + +![Maintenance window overview](./images/alarm-maintenance-window-1.png) + +## Creating a maintenance window + +1. Navigate to the **Maintenance Windows** page in the Alerts module. +2. Click **Create Maintenance Window**. + +![Create Maintenance Window](./images/alarm-maintenance-window-1.jpeg) + +3. Configure these details: + +- **Name** — Name for the maintenance window. +- **Description** — Short description of the maintenance window. +- **Timezone** — Timezone for the scheduled maintenance window. +- **One time** — A one-time maintenance window schedules a maintenance period only once for a specific date and time. +- **Recurring** — A recurring maintenance window automatically repeats on a regular cycle, eliminating the need to configure it each time. + +![Maintenance window timing options](./images/alarm-maintenance-window-2.jpeg) + +Additional settings: + +- **Start time** — Date and time when the maintenance window begins. +- **End time** — Date and time when the maintenance window ends. +- **Frequency** — Available only for recurring windows; select daily, weekly, or monthly schedules. +- **Label Matchers** — Target the window to specific alerts by their labels (e.g. service name, host, region). Each matcher is a label key, an operator, and a value, combined with AND. The operators are the same four used across the alert module — **Equals**, **Not equals**, **Matches regex**, **Doesn't match regex** — matched against alert rule labels and alert instance labels. + +4. Click **Save** to create the new maintenance window. + +![Maintenance window saved](./images/alarm-maintenance-window-3.jpeg) diff --git a/src/content/docs/docs/alerts/routing-rules.mdx b/src/content/docs/docs/alerts/routing-rules.mdx new file mode 100644 index 00000000..cf3b71c0 --- /dev/null +++ b/src/content/docs/docs/alerts/routing-rules.mdx @@ -0,0 +1,137 @@ +--- +title: "Routing Rules" +description: "Route alert notifications to the right channels based on label matchers and grouping." +sidebar: + order: 5 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Routing Rules decide which channels get notified for which alerts. They govern **every** alert notification — both **individual alerts** and **correlated alert groups**. Each rule matches alerts by their labels; the **first matching rule wins** and sends those alerts to its destination channels. Whether the matched alerts notify one-by-one or fold into a single [Alert Group](../alert-groups/) is decided by the rule's **group-by** keys. + +:::note[Not the incident routing rules] +This page routes alert *notifications* to channels. To control how an alert becomes an incident inside Incident Management — its service, escalation policy, and severity — see [Incident Management → Routing Rules](../../incident-management/routing-rules/) instead. +::: + +Routing Rules replace the older **Notification Policies** flow. If you previously routed by tags, your existing policies were migrated to routing rules during upgrade — the matching logic now keys off [labels](../create-alerts/#4-folder-severity-and-labels) instead of free-form tags. + +## Where to find them + +Open **Alerts → Routing rules** in the left navigation. Rules are evaluated in priority order (lowest number wins). The first rule an alert matches decides which channels notify and how often — and, through its group-by keys, whether the alert notifies on its own or correlates with others into one [Alert Group](../alert-groups/). + +![Routing Rules list](./images/routing-rules-list.png) + +The list shows every routing rule in the workspace, sorted by priority (lower wins). The **default passthrough rule** sits at the bottom — it matches everything that didn't already match a higher-priority rule and can't be deleted. + +## Anatomy of a rule + +A routing rule has six parts: + +| Field | What it does | +|---|---| +| **Name** | How the rule appears in the list and audit log. | +| **Priority** | Lower numbers win. When an alert matches multiple rules, the highest-priority (lowest-numbered) rule routes it. | +| **Matchers** | Label conditions that pick which alerts this rule applies to. | +| **Group-by** | Whether and how matched alerts correlate. Leave it empty (or set only `alarm_id`) for **one notification per alarm rule** — no grouping. Add a real label key (e.g. `service`) to **correlate** matching alerts that share that value into one [Alert Group](../alert-groups/). `severity` can't be used here. | +| **Cadence** | `group_wait` (how long to wait after the first signal before sending the first notification) and `group_interval` (how often to re-notify on an open group). | +| **Destination channels** | One or more notification channels that receive the dispatched payload. | + +Optional: + +- **Auto-RCA** — toggle to run KloudMate's AI investigator on every group this rule opens. See [Auto-RCA](../auto-rca/). + +### Matchers + +Each matcher is one label condition — a label key, an operator, and a value. An alert has to satisfy **every** matcher you add (they combine with AND) before this rule handles it. Leave the list empty and the rule becomes a catch-all: it handles every alert no higher-priority rule already caught. + +The available operators: + +| Operator | Wire format | Meaning | +|---|---|---| +| Equals | `equals` | Label value matches the string exactly. | +| Not equals | `not_equals` | Label value differs from the string (also matches alerts that don't carry the label at all). | +| Matches regex | `matches_regex` | Label value matches the regular expression. | +| Doesn't match regex | `not_matches_regex` | Label value doesn't match the regex (also matches alerts without the label). | + +A matcher value is always a single string. To match any of several values, use a regex alternation — for example `service` **Matches regex** `api|web` matches either service. (There's no `in`/`not_in` operator; the regex form replaces it.) + +As you add or edit matchers, the editor shows a live **Affected Alert Instances** preview — the alert instances this rule currently matches, with a running match count — so you can confirm the rule's scope before saving. + +### Group-by keys + +Group-by decides **whether** matched alerts correlate and, if so, along **which** dimension: + +- **Leave it empty — or set only `alarm_id`** — and the rule doesn't correlate anything: each alarm rule notifies on its own, one notification per rule. These notifications surface through the normal alerts UI, not the [Alert Groups](../alert-groups/) page. +- **Add a real label key** and matching alerts that share that key's value fold into a single alert group. The editor suggests common keys — `service`, `env`, `host`, `region`, `team`, `issue_id`, `check_id`, `alarm_rule_folder_id` — and you can type any other label key. + +Two constraints the editor enforces: + +- If you add a real dimension alongside `alarm_id`, the `alarm_id` is dropped automatically — pinning each group to a single alarm would defeat the correlation. +- `severity` can't be a grouping dimension. It's a per-instance property fixed when the group opens, not an axis to group on. + +You can combine keys (e.g. `service` + `env`) to scope each group tightly. Pick keys based on which dimension changes the most: aggressive grouping (only `service`) reduces notification noise but sacrifices precision in the group title. + +### Cadence + +- **group_wait** — default `30s`. The grouping engine holds the first notification for this long after a group opens, so closely related signals fold in before paging. +- **group_interval** — default `5m`. Once a group is open, KloudMate re-notifies at this cadence if new signals continue to arrive. + +## The default passthrough rule + +Every workspace has a default passthrough rule pinned to the bottom of the list. It matches everything that didn't match a higher-priority rule. You can edit its destination channels and cadence, but you can't delete it or change its matchers. + +If you want a clean "everything else goes to Slack" path, point the default passthrough at your fallback Slack channel. + +## Create a routing rule + + +1. **Open Alerts → Routing rules** and click **Create rule**. + +2. **Name the rule** and set its priority. Lower numbers win — give specific rules low numbers (10, 20, 30) and generic ones higher numbers. The default passthrough sits at priority 1,000,000 so unmatched alerts always have somewhere to land. + +3. **Add matchers.** Pick a label key, an operator, and a value. Add as many rows as needed; all matchers are combined with AND. + +4. **Set group-by keys.** Leave this empty to notify once per alarm rule, or add a real label key (start with `service`) to correlate matching alerts into one group. `severity` isn't allowed here. + +5. **Tune cadence.** Leave the defaults (`30s` wait, `5m` interval) for most cases. Tighten them for paging rules, or loosen them for digest-style summaries. + +6. **Pick destination channels.** Add one or more channels. If you don't have a [KloudMate Incidents](../../platform/settings/notification-channels/#kloudmate-incidents) channel yet, use the inline **+ Add KloudMate Incidents channel** link in the picker. + +7. **Optionally enable Auto-RCA.** Toggle Auto-RCA on and set the delay (default 5 minutes after group open). See [Auto-RCA](../auto-rca/). + +8. **Save.** The rule appears in the list at its assigned priority. + + +## Row actions + +The kebab menu on each row offers: + +- **Edit** — opens the rule editor. +- **Duplicate** — opens the editor pre-filled with the rule's settings under a "(copy)" name so you can adjust matchers or priority and save as a new rule. +- **Delete** — removes the rule. The delete is **blocked while open alert groups still route through this rule** — resolve or re-route those groups first, then delete. The default passthrough rule can't be deleted at all. + +The **Enabled** toggle in each row disables the rule without deleting it — useful for staging a rule before enabling it. + +## Suggest rules + +The list header has a **Suggest rules** button (visible to admins only). Clicking it opens a drawer with up to five AI-suggested rules based on your workspace's recent alert traffic. + +![Suggest rules drawer](./images/routing-rules-suggest.png) + +Each suggestion shows: + +- A title describing the proposed grouping (e.g. "Group by service + env"). +- A short rationale — how much notification noise the rule would have collapsed over the lookback window. +- A **preview matches** count for that window. +- **Accept** opens the create-rule form pre-filled with the suggestion's matchers, group-by keys, and cadence, under a name derived from what the rule targets. **Dismiss** drops the card. + +Some suggestions are grouping-only: they propose group-by keys with no matchers. Accepting one starts the rule as a catch-all that groups everything by those keys — add matchers before saving if you want to scope it to a subset of alerts. + +Suggestions don't persist — closing the drawer discards them. Re-run the suggester whenever you want fresh proposals. + +## Related + +- [Alert Groups](../alert-groups/) — what routing rules produce. +- [Silences](../silences/) — temporarily suppress matching alerts. +- [Notification Channels](../../platform/settings/notification-channels/) — set up Slack, KloudMate Incidents, email, and more. +- [Auto-RCA](../auto-rca/) — automatic investigations on group open. diff --git a/src/content/docs/docs/alerts/silences.mdx b/src/content/docs/docs/alerts/silences.mdx new file mode 100644 index 00000000..1a1ce902 --- /dev/null +++ b/src/content/docs/docs/alerts/silences.mdx @@ -0,0 +1,76 @@ +--- +title: "Silences" +description: "Suppress alert notifications for matching labels for a bounded time window." +sidebar: + order: 7 +--- + +A **Silence** suppresses notifications for alerts whose labels match the silence's matchers, for a bounded time window. Use silences when you know an alert will be noisy and you don't want it paging anyone — a deployment expected to spike error rates, a known third-party outage, or a maintenance window you can't fully scope. + +Silences cap at 30 days. The cap exists so silences don't get forgotten about and quietly mask real problems — re-create one explicitly if you need it longer. + +## Where to find silences + +Open **Alerts → Silences** in the left navigation. Silences are ad-hoc — for recurring quiet windows use [Maintenance Windows](../maintenance-windows/) instead. + +![Silences list](./images/silences-list.png) + +The list has three tabs with live counts: **Active (N)** (default), **Expired (N)**, and **All (N)**. Columns: **Matchers** (chips, with overflow tooltip), **Expires** (countdown like e.g. *"In 4h"* or *"Expired 12m ago"*), **Scope** (a `Standalone` or `Group-bound` chip), **Reason**, **Created by**. + +## Anatomy of a silence + +| Field | What it does | +|---|---| +| **Matchers** | Label conditions that pick which alerts the silence applies to. Same operator set as [Routing Rules](../routing-rules/#matchers): Equals, Not equals, Matches regex, Doesn't match regex. | +| **Expires at** | Date and time the silence stops. Capped at 30 days from creation. | +| **Reason** | Free-form text — recommended so others know why the silence exists. | +| **Scope** | Either **Group-bound** (auto-expires when a specific [Alert Group](../alert-groups/) resolves) or **Standalone** (lives until its `expires_at`). | + +## Create a silence + +There are a few ways to create a silence: + +### From Alerts → Silences + +1. Open **Alerts → Silences** and click **New silence**. +2. Add one or more matcher rows. For each: pick a label key, an operator, and a value. +3. Pick an **Expires at** date/time. The form caps duration at 30 days and disables submit otherwise. +4. Add a **Reason** describing why this silence exists. +5. Click **Save**. The silence appears in the **Active** tab. + +### From an Alert Group + +On any [Alert Group](../alert-groups/) detail page, click **Silence this group**. The silence creator opens pre-filled with: + +- The group's labels populated as matchers. +- A hidden `auto_expire_group_id` set to the current group, so the silence stops automatically when the group resolves. +- A banner above the form reads e.g. "This silence will auto-expire when group `` resolves." + +Save it from here. The silence shows up as **Group-bound** on the list. + +### From an alert's Pause Notifications + +On an alert rule's **more options (⋯) → Pause Notifications**, KloudMate creates a silence scoped to that alert (an `alarm_id` matcher) with an expiry — the alert keeps evaluating, only its notifications are suppressed. While paused, the rule is marked **Silenced** on the alerts list and its detail page, and the silence appears here in the **Active** tab, where you view or end it. Add matchers to narrow the pause to specific instances. + +## Silence vs. maintenance window + +Both features are a **notification gate, not a state change** — the alert still evaluates on its schedule and its state transitions are still recorded in history; only the notification is withheld. Silencing every firing instance of an alert group keeps the group **Open** (shown as **Muted**) — it does **not** resolve the group, and muting never emits a *resolved* notification. The differences between the two are operational, not behavioral: + +- **Silences** are ad-hoc, label-matcher driven, capped at 30 days, and can be auto-bound to a specific alert group so they expire when the group resolves. +- **Maintenance Windows** are scheduled — one-time (RFC3339 start/end) or recurring (`DAILY` / `WEEKLY` / `MONTHLY` with a timezone-aware time-of-day). + +Both use the same four label-matcher operators — **Equals**, **Not equals**, **Matches regex**, **Doesn't match regex** — matched against alert rule labels and alert instance labels. + +Both leave the alert evaluating. Suppressed state transitions are marked with `silenced_at` and `silenced_by_*` columns visible on the alert state history, so you can always tell after the fact whether a quiet stretch was a real recovery or a suppressed notification. + +Pick a silence for a reactive, one-off suppression you'll re-evaluate within 30 days. Pick a [Maintenance Window](../maintenance-windows/) for planned downtime, recurring quiet hours, or anything that needs a calendar schedule. + +## Viewing and ending a silence + +Each row's kebab menu has **View** and **End silence**. Ending an active silence removes suppression for matching alerts immediately — new firings notify as normal. + +## Related + +- [Alert Groups](../alert-groups/) — Silence-this-group action. +- [Maintenance Windows](../maintenance-windows/) — scheduled, recurring counterpart to silences. +- [Routing Rules](../routing-rules/) — same matcher model. diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx new file mode 100644 index 00000000..a42c099e --- /dev/null +++ b/src/content/docs/docs/apm-and-tracing/apm-views/api-monitoring.mdx @@ -0,0 +1,110 @@ +--- +title: "API Monitoring" +description: "Use API Monitoring to see Rate, Errors, and Duration (RED) metrics for every HTTP and RPC endpoint, then drill into an endpoint's trends and traces." +sidebar: + order: 4 +--- + +**API Monitoring** shows Rate, Errors, and Duration (RED) metrics for every API endpoint your services handle, across both HTTP and RPC (gRPC and Connect / `connect_rpc`). It is the server-side, per-endpoint view of your traffic. + +Where the [Service Map](../service-map/) shows caller-to-callee dependencies and the [Trace Explorer](../../trace-explorer/) helps you find an individual request, API Monitoring sits in between: it answers "which of my endpoints are slow, failing, or busy?" and lets you drill from a single endpoint straight into its traces. + +## Overview + +In the left navigation, open **APM & Tracing → API Monitoring**. + +:::note[Screenshot placeholder] +Add a screenshot of the API Monitoring page here (`./images/api-monitoring-1.png`): the page header, the filter bar, and the endpoints table with protocol badges, the Status column, and throughput sparklines visible. +::: + +Use the **time range** picker in the top-right to set the observation window. The view defaults to the last **24 hours** so the per-endpoint trend sparklines have enough history to draw. + +Endpoints are derived from your services' server spans. If the table is empty, confirm your services are instrumented and sending traces, or widen the time range — see [Getting Accurate Data](#getting-accurate-data). + +## The Endpoints List + +Each row is one endpoint, identified by its service, protocol, and (for HTTP) method. The columns are: + +- **Endpoint:** a protocol badge (`HTTP`, `GRPC`, or `CONNECT_RPC`), a method chip for HTTP rows (`GET`, `POST`, …), and the endpoint label — a templated route such as `GET /users/{id}` for HTTP, or the full method name such as `pkg.Svc/Method` for RPC. +- **Service:** the service that handles the endpoint. It links to that service's [APM dashboard](../apm-and-services/). +- **Throughput (req/min):** request rate over the selected window. +- **Error rate:** the percentage of requests counted as errors (see [How Errors Are Counted](#how-errors-are-counted)). Shown in red when above zero. +- **p50 / p95 / p99:** latency percentiles, in milliseconds. +- **Status:** an error-focused summary of the HTTP status mix. It reads `healthy` when there are no `4xx` or `5xx` responses, and surfaces `4xx` and `5xx` counts when they occur. Hover it to see the full `2xx` / `3xx` / `4xx` / `5xx` breakdown, plus any uncategorized requests. +- **Throughput trend:** a sparkline of request volume across the window. + +### HTTP vs. RPC rows + +The protocol badge determines what a row can show: + +- **HTTP** rows carry a method chip and a full `2xx`/`3xx`/`4xx`/`5xx` status breakdown. +- **RPC** rows (gRPC, Connect) have no HTTP method or status classes, so the **Status** column shows a dash (`—`). Judge these endpoints by their **error rate**, which is derived from span status and works for every protocol. + +:::tip +The trend sparkline rolls up hourly, so it needs at least two hourly buckets to draw a line. +::: + +## Filtering and Sorting + +Use the filter bar above the table to narrow the list: + +- **Search:** match endpoints by name (for example, a path fragment or RPC method). +- **Service:** one or more services. +- **Protocol:** `http`, `grpc`, or `connect_rpc`. +- **Method:** one or more HTTP methods (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`). +- **Status:** restrict to endpoints that returned a given HTTP status class — `4xx` or `5xx`. + +Click the **Throughput**, **Error rate**, or **p95** column header to sort by that metric. Sorting is always highest-first and is applied across all matching endpoints, so the table always shows the true top results rather than re-ordering only the current page. + +:::tip +To find failing endpoints quickly, sort by **Error rate**, or set the **Status** filter to `5xx`. +::: + +## Endpoint Details + +Click any row to open the endpoint drill-down in a slide-over panel. The panel is shareable — its address encodes the selected endpoint, so you can send a teammate a direct link, and closing it returns you to the list with your filters, sort, and time range intact. + +:::note[Screenshot placeholder] +Add a screenshot of the endpoint drill-down here (`./images/api-monitoring-2.png`): the drawer header with protocol and method badges, the Requests / Error rate / p95 summary tiles, and the time-series charts. +::: + +At the top, summary tiles show **Requests**, **Error rate**, and **p95 latency** for the endpoint over the selected window. Below them are time-series charts: + +- **Throughput (requests)** over time. +- **Error rate** over time. +- **Latency** with `p50`, `p95`, and `p99` overlaid. +- **Status classes** (`2xx` / `3xx` / `4xx` / `5xx`) over time — shown for HTTP endpoints only. + +### Recent and Failed Traces + +The drill-down ends with a table of the endpoint's traces, so you can move from a metric to the exact requests behind it: + +- Each row shows the start time, span, and duration, with a **View trace** action that opens the full trace in [Trace Detail](../../trace-detail/). +- Toggle **Failed only** to keep just the errored requests. This uses the same protocol-neutral error definition as the metrics, so it covers HTTP `5xx` and gRPC/Connect errors alike. +- **Open in Traces** hands the current endpoint and time range to the [Trace Explorer](../../trace-explorer/) for deeper filtering and analysis. + +## How Errors Are Counted + +An endpoint **error** is any request that either returned an HTTP **`5xx`** status **or** produced a span marked `ERROR` (an exception, or a gRPC/Connect error). This definition is protocol-neutral, which is why RPC endpoints get a meaningful error rate even though they have no HTTP status. + +**`4xx` responses are treated as client faults, not endpoint errors.** They appear in the status breakdown but are deliberately excluded from the error rate, so a spike in client mistakes (bad input, unauthorized calls) does not make a healthy endpoint look broken. + +:::note +API Monitoring is a **server-side** view: it counts what each endpoint actually returned. A *caller-perceived* failure — for example, a client that times out while calling an otherwise healthy service — does not appear here. Look for those on the corresponding [Service Map](../service-map/) edge, which measures the call from the caller's side. +::: + +## Getting Accurate Data + +API Monitoring is built entirely from your tracing data. For complete and well-grouped metrics: + +- Services must emit OpenTelemetry **server spans** (`span.kind = server`). Each span needs HTTP attributes (method and status code) for HTTP endpoints, or `rpc.system` for gRPC/Connect endpoints. +- For clean grouping, routes should be **templated by the instrumentation** — `GET /users/{id}`, not `GET /users/12345`. Untemplated routes with raw IDs would otherwise create a separate endpoint per value. KloudMate auto-collapses obvious numeric and UUID path segments as a safety net, but SDK route templates give the best result. +- **gRPC and Connect** endpoints are grouped by their full method name and work out of the box — no templating needed. + +If you are still setting up instrumentation, see the [Auto Instrumentation](../../auto-instrumentation/) and [Manual Instrumentation](../../manual-instrumentation/) guides. + +## Related Paths + +- [Service Map](../service-map/) — service-to-service dependencies and the caller's view of failures. +- [Services](../apm-and-services/) — service-level request, latency, and error trends. +- [Trace Explorer](../../trace-explorer/) — search and filter individual traces. diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/apm-and-services.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/apm-and-services.mdx index 47edaa2f..07ed8942 100644 --- a/src/content/docs/docs/apm-and-tracing/apm-views/apm-and-services.mdx +++ b/src/content/docs/docs/apm-and-tracing/apm-views/apm-and-services.mdx @@ -32,7 +32,7 @@ The dashboard includes: - **Requests, errors, and latency over time:** timeline charts that help identify traffic spikes, error bursts, and latency regressions - **Destinations table:** a breakdown of downstream dependencies, including request rate, error percentage, and dependency latency -- **Related issues, alarms, and dashboards:** links to broader operational context for that service +- **Related issues, alerts, and dashboards:** links to broader operational context for that service ## Traces and Logs @@ -43,3 +43,5 @@ The **Traces** table shows recent traces with details such as start time, span n ![image](./images/apm-and-services-3.png) The **Logs** table lists recent logs with time, severity, and message, along with actions to explore logs in detail and correlate them with traces and issues for faster root-cause investigation. + +![image](./images/apm-and-services-4.png) diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-1.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-1.png index 3e0ca5ac..d0265cd9 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-1.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-1.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-2.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-2.png index ea26b9e9..0049e6f5 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-2.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-2.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-3.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-3.png index 6766aa98..293ad813 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-3.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-3.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-4.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-4.png new file mode 100644 index 00000000..787dc894 Binary files /dev/null and b/src/content/docs/docs/apm-and-tracing/apm-views/images/apm-and-services-4.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.jpeg b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.jpeg index 0c990c51..156d4b82 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.jpeg and b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.png index 316c3498..47a9656f 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-1.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.jpeg b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.jpeg index 5845bcc0..367e2503 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.jpeg and b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.png index 61b526bb..285c428c 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-2.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-3.png b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-3.png index 69c0ee62..cc3ab7c1 100644 Binary files a/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-3.png and b/src/content/docs/docs/apm-and-tracing/apm-views/images/service-map-3.png differ diff --git a/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx b/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx index ef7e53ad..3f483ae6 100644 --- a/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx +++ b/src/content/docs/docs/apm-and-tracing/apm-views/service-map.mdx @@ -30,7 +30,7 @@ Click any service node on the graph to open the service details panel on the rig ![image](./images/service-map-1.jpeg) -Click **Open service page** at the bottom of the panel to navigate to that service's APM dashboard, where you can inspect request trends, latency percentiles, traces, issues, alarms, and dashboards in more detail. +Click **Open service page** at the bottom of the panel to navigate to that service's APM dashboard, where you can inspect request trends, latency percentiles, traces, issues, alerts, and dashboards in more detail. ![image](./images/service-map-3.png) diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-1.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-1.jpeg index 768a629d..fee42d15 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-1.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-1.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-1.png b/src/content/docs/docs/apm-and-tracing/images/traces-1.png index e7f91682..e42c73dc 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-1.png and b/src/content/docs/docs/apm-and-tracing/images/traces-1.png differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-2.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-2.jpeg index 8437b7c9..4f27569c 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-2.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-2.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-2.png b/src/content/docs/docs/apm-and-tracing/images/traces-2.png index a66e18e3..84e71c2c 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-2.png and b/src/content/docs/docs/apm-and-tracing/images/traces-2.png differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-3.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-3.jpeg index bc948f8c..da565b70 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-3.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-3.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-4.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-4.jpeg index 9a12faa7..0c67495f 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-4.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-4.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-5.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-5.jpeg index 548da7a2..70ba6c96 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-5.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-5.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/images/traces-6.jpeg b/src/content/docs/docs/apm-and-tracing/images/traces-6.jpeg index 768a629d..88cd8d1b 100644 Binary files a/src/content/docs/docs/apm-and-tracing/images/traces-6.jpeg and b/src/content/docs/docs/apm-and-tracing/images/traces-6.jpeg differ diff --git a/src/content/docs/docs/apm-and-tracing/index.mdx b/src/content/docs/docs/apm-and-tracing/index.mdx index 41f548bc..51e30f4f 100644 --- a/src/content/docs/docs/apm-and-tracing/index.mdx +++ b/src/content/docs/docs/apm-and-tracing/index.mdx @@ -13,9 +13,10 @@ The `APM & Tracing` section in KloudMate gives you end-to-end visibility into ho ## Core Concepts - **[APM Views](./apm-views/)** is service-centric. Use it to compare latency, throughput, error rate, and dependencies across services. -- **[Trace Explorer](./trace-explorer/)** is request-centric. Use it to debug a specific request, span, or execution path across multiple services. +- **[Trace Explorer](./trace-explorer/)** is request-centric. Use it to search, filter, and find a specific trace. +- **[Trace Detail](./trace-detail/)** is where you inspect one trace — the timeline waterfall, per-span detail, and the Service List, Request Flow, and AI Flow views. -Start in **APM Views** when you need higher-level service health and dependency analysis. Navigate to **Trace Explorer** when you need to inspect trace-level detail to find the root cause of an issue. +Start in **APM Views** when you need higher-level service health and dependency analysis. Move to **Trace Explorer** to find the trace you care about, then open it in **Trace Detail** to find the root cause. ## Instrumentation Paths @@ -31,5 +32,6 @@ Because Application Metrics and Custom Metrics are emitted through the exact sam ## Related Paths +- [LLM Observability](../llm-observability/) - [Visualize Data](../visualize-data/) - [Logs](../logs/) diff --git a/src/content/docs/docs/apm-and-tracing/trace-detail.mdx b/src/content/docs/docs/apm-and-tracing/trace-detail.mdx new file mode 100644 index 00000000..c81cbdde --- /dev/null +++ b/src/content/docs/docs/apm-and-tracing/trace-detail.mdx @@ -0,0 +1,64 @@ +--- +title: "Trace Detail" +description: "Inspect a single trace in KloudMate — the timeline waterfall, span detail panel, and the Service List, Request Flow, and AI Flow views." +sidebar: + order: 2 +--- + +Open a trace from the [Trace Explorer](/apm-and-tracing/trace-explorer/) to inspect a single request end to end. The detail view lays out the whole trace: a summary on the left, a chosen view on the right, and the timeline along the bottom. Use it to find which span was slow, which one errored, and why. + +## Trace summary + +Until you select a span, the left panel shows the trace summary: + +- **Trace ID, date, and total duration** for the request. +- **Per-service time** — each service in the trace with its share of the request time, so you can see where the request spent most of its time. +- **Errors** — any errored spans, with a link to jump straight to the span. + +When the trace contains GenAI spans, an **AI summary** appears here too — models, model and tool call counts, token usage, and the model-versus-tool time split. See [AI Trace Observability](/llm-observability/ai-trace-observability/). + +## Timeline + +The timeline along the bottom is the waterfall of spans, ordered by start time and nested by parent. Each row shows the span's service, operation name, an icon for its type, and a duration bar positioned by when it ran. + +- **Expand or collapse** a span to show or hide its children, or use **Toggle Expand All**. +- **Hide internal spans** to cut framework noise and focus on the spans that cross a boundary. +- **Search within the trace** using the filter box. Matching spans stay highlighted while the rest dim, and the match counter lets you step through hits with the next and previous arrows. + +Select any row to open its detail in the left panel. + +## Span detail + +Selecting a span replaces the summary with the span's detail, organized into tabs. The tabs you see depend on the span's type, so you only get what's relevant: + +- **Request** and **Response** for HTTP spans — method, route, status, headers, and body. +- **Database** for database spans — the statement and database attributes. +- **Message** for messaging spans. +- **Conversation** and **Gen AI** for LLM spans — the prompt and response transcript, token usage, finish reason, and tool calls. See [AI Trace Observability](/llm-observability/ai-trace-observability/). +- **Info** for service, host, Kubernetes, and deployment metadata. +- **Events** and **Links** when the span has them. +- **Attributes** for the full attribute set, grouped by semantic-convention prefix. +- **JSON** for the raw span. + +## Views + +The toggle at the top right switches how the right panel presents the trace: + +- **Service List** lists the spans in a table — operation, type, and duration — for a flat, scannable view of everything in the trace. +- **Request Flow** draws the trace as a service graph, so you can see how services called each other and where errors propagated. +- **AI Flow** appears for traces with GenAI spans. It lays out the model and tool steps as a workflow graph. See [AI Trace Observability](/llm-observability/ai-trace-observability/). + +Selecting a span in any view — or in the timeline — keeps the detail panel in sync. + +## Request logs and assistant + +Two shortcuts sit in the trace header: + +- **Request Logs** opens the logs correlated with this trace, when logs are available, so you can move from a span to the log lines around it. +- **Ask Assistant** hands the trace to the KloudMate assistant to help investigate bottlenecks or errors. + +## Related paths + +- [Trace Explorer](/apm-and-tracing/trace-explorer/) +- [APM Views](/apm-and-tracing/apm-views/) +- [AI Trace Observability](/llm-observability/ai-trace-observability/) diff --git a/src/content/docs/docs/apm-and-tracing/trace-explorer.mdx b/src/content/docs/docs/apm-and-tracing/trace-explorer.mdx index 7f4673e2..f7527d3f 100644 --- a/src/content/docs/docs/apm-and-tracing/trace-explorer.mdx +++ b/src/content/docs/docs/apm-and-tracing/trace-explorer.mdx @@ -5,7 +5,7 @@ sidebar: order: 1 --- -Click **Traces** in the left navigation menu to open the section. +Click **Traces** in the left navigation menu to open the section. It opens on the **Trace Summary** view by default. At the top of the page, the **span volume graph** visualizes trace activity over the selected time range. This gives you a quick pulse check on traffic patterns before you inspect individual traces. @@ -32,10 +32,11 @@ Click the filter icon on the right to open the full attribute picker. Active fil Use the left sidebar to filter results by: - Service +- Severity - HTTP Method - HTTP Status -- Severity - Database Name +- Duration (a latency histogram with a range slider) Each filter section includes a search box for quick lookup. When filters are active, a recent-filters notice appears at the top of the sidebar. @@ -49,33 +50,50 @@ Click **Saved Queries** to view and apply previously saved searches. To save a n ## Traces Result Views -### Spans +Switch between views using the toggle above the results. They all read the same trace data through the active time range and filters — each gives you a different level of detail. -The **Spans** tab is the default view. It lists every individual span captured within the selected time range and active filters. Each row displays `Timestamp`, `Span Name`, `Services`, `Duration`, and `Error`. +### Trace Summary -This view is best when you need the most granular view of tracing activity. +**Trace Summary** is the default view. It groups spans into complete traces, giving you a higher-level view of end-to-end request journeys. Each row represents a single trace, showing its entry point, number of services, spans involved, total duration, and any errors. -![image](./images/traces-6.jpeg) +![image](./images/traces-2.png) -### Trace Summary +### Trace Groups -The **Trace Summary** tab groups spans into complete traces, giving you a higher-level view of end-to-end request journeys. Each row represents a single trace, showing its entry point, number of services, spans involved, total duration, and any errors. +**Trace Groups** rolls traces up by **entry point** — one row per `(service, root operation)` — so you can see which endpoints drive your traffic, latency, and errors at a glance. Each row shows: -![image](./images/traces-2.png) +- **Entry point** — the root service and operation +- **Traces** — number of traces for that entry point +- **Spans (avg)** — average spans per trace +- **Entities (avg)** — average number of services involved per trace (hover for the per-service breakdown) +- **p95** — 95th-percentile trace duration +- **Errors** — number of traces with errors +- **Trend** — a sparkline of trace volume over the window (hover for per-bucket counts) + +Filter the groups from the left panel by **entry operation**, **service**, and **duration**. -### Root Spans +#### Drilling into a group -The **Root Spans** tab shows only the root span from each trace. This makes it easier to compare top-level requests without the noise of child spans. +Click a group row to open its **drill-down drawer**. The drawer shows the group's metrics across the top, then the list of traces that belong to it. From here you can: -![image](./images/traces-3.png) +- **Filter the group's traces** by span attributes using the drawer's search bar. +- **Scroll the list** — it pages automatically as you reach the bottom. +- **Open any trace** — click a row to open its full [Trace Detail](../trace-detail/) stacked on top of the drawer. Closing it returns you to the group; closing the group returns you to the list. + +### Spans -### Services +The **Spans** view lists every individual span captured within the selected time range and active filters. Each row displays `Timestamp`, `Span Name`, `Services`, `Duration`, and `Error`. + +This view is best when you need the most granular look at tracing activity. To isolate errors, add a **Severity → Error** filter from the sidebar. + +![image](./images/traces-6.jpeg) -The **Services** tab summarizes traffic by service so you can identify hotspots, dependency trends, and the applications generating the most trace volume. +## Open a trace -![image](./images/traces-4.png) +Select a trace or span from any result view to open it in [Trace Detail](../trace-detail/), where you inspect the timeline waterfall, per-span attributes, and the Service List, Request Flow, and AI Flow views. ## Related Paths +- [Trace Detail](../trace-detail/) - [APM Views](../apm-views/) - [Explore](../../visualize-data/explore/) diff --git a/src/content/docs/docs/aws-integration/index.mdx b/src/content/docs/docs/aws-integration/index.mdx index e1f54091..2ec721e3 100644 --- a/src/content/docs/docs/aws-integration/index.mdx +++ b/src/content/docs/docs/aws-integration/index.mdx @@ -19,6 +19,7 @@ Once connected, KloudMate can automatically discover AWS resources and begin ing - **Automatic Resource Discovery**: Quickly discover and monitor supported AWS resources, starting with AWS Lambda. - **Centralized CloudWatch Logs**: Subscribe to existing CloudWatch Log Groups directly from the KloudMate dashboard and forward them into the centralized Log Explorer. - **Serverless Observability**: Get out-of-the-box dashboards and metrics (such as invocations, errors, cold starts, and memory usage) for AWS Lambda workloads. +- **Direct Lambda Telemetry**: Push logs, metrics, and traces straight from a function with the [Lambda Telemetry Extension](./lambda-telemetry-extension/) layer — no CloudFormation stack or CloudWatch subscription required. ## Available Guides @@ -27,3 +28,4 @@ Explore the following guides to set up and configure your AWS integrations: - **[AWS Account Setup](./account-setup/)**: Learn how to connect your AWS environment to KloudMate using CloudFormation and cross-account IAM roles. - **[CloudWatch Logs](./cloudwatch-logs/)**: Instructions for subscribing to CloudWatch Log groups and making them fully searchable inside KloudMate. - **[AWS Lambda Monitoring](./lambda-monitoring/)**: Explore how KloudMate aggregates Lambda metrics, invocations, and issues in a unified console. +- **[Lambda Telemetry Extension](./lambda-telemetry-extension/)**: Send Lambda logs, metrics, and traces to KloudMate with a layer, skipping the cross-account CloudFormation setup. diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/_meta.json b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/_meta.json new file mode 100644 index 00000000..0561f608 --- /dev/null +++ b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/_meta.json @@ -0,0 +1,5 @@ +{ + "label": "Lambda Telemetry Extension", + "order": 5, + "collapsed": true +} diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/configuration.mdx b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/configuration.mdx new file mode 100644 index 00000000..01a984cb --- /dev/null +++ b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/configuration.mdx @@ -0,0 +1,74 @@ +--- +title: "Configuration & Data" +description: "Environment variables for the KloudMate Lambda Telemetry Extension and the metrics it sends." +sidebar: + label: "Configuration & Data" + order: 4 +--- + +Set these environment variables on the Lambda function, alongside the layer. For the full walkthrough, start with [Setup](../setup/). + +## Required + +| Variable | Value | +| --- | --- | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | Your KloudMate OTLP endpoint, for example `https://otel.kloudmate.com:4318`. | +| `OTEL_EXPORTER_OTLP_HEADERS` | Your ingest key as an `Authorization` header: `Authorization=YOUR_KLOUDMATE_API_KEY`. Create a key under [API Keys](../../../platform/settings/api-keys/). | + +## Optional + +| Variable | Value | +| --- | --- | +| `OTEL_SERVICE_NAME` | The name your function appears under in KloudMate. Defaults to the function name. | +| `OTEL_LOGS_EXPORTER` | Set to `none` to stop sending logs — for example, when the [AWS account integration](../../account-setup/) already collects this function's logs from CloudWatch. | +| `OTEL_METRICS_EXPORTER` | Set to `none` to stop sending metrics. | +| `OTEL_TRACES_EXPORTER` | Set to `none` to stop sending traces. | +| `OTEL_LOG_LEVEL` | Set to `debug` to print the extension's own diagnostics to CloudWatch — useful when telemetry isn't showing up. | + +## Metrics + +The extension reports these for every invocation. Use the names to build dashboards and alerts in KloudMate. + +| Metric | Unit | What it measures | +| --- | --- | --- | +| `faas.invoke_duration` | seconds | Total invocation time | +| `faas.init_duration` | seconds | Cold-start time (cold starts only) | +| `faas.mem_usage` | bytes | Peak memory used | +| `aws.lambda.billed_duration` | seconds | AWS billed time | +| `aws.lambda.produced_bytes` | bytes | Response payload size | +| `aws.lambda.runtime_duration` | seconds | Handler time, not counting the extension | +| `aws.lambda.restore_duration` | seconds | SnapStart restore duration | + +Each metric is tagged with the request ID, a cold-start flag, and the invocation status, so you can filter and group by any of them. + +You can use these metrics to build dashboards that track your function's health and resource consumption over time. + +![Lambda OTel Dashboard showing invocation trends, memory usage, billed duration, and cold start metrics](./images/telemetry-metrics-dashboard.png) + +## Logs and traces + +Your function's stdout/stderr lines arrive as logs, with the severity read from each line. Without active tracing, you still get logs and metrics. + +### Logs in Log Explorer + +Every logs or print statement from your function code is forwarded directly to the **Log Explorer**, alongside lifecycle events like `START`, `END`, and `REPORT` requests. + +![Log Explorer displaying Lambda execution logs and lifecycle messages](./images/telemetry-log-explorer.png) + +### Trace Correlation + +With [X-Ray active tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html) enabled, each invocation also produces a trace. This connects the logs from that specific execution to a unified trace timeline. + +In the **Traces** explorer, you can list and filter invocations by their duration and status: + +![Traces list showing recent execution spans and durations](./images/telemetry-traces-list.png) + +Selecting an execution trace opens a detailed breakdown of the execution timeline. This breaks down the lifecycle into sub-spans like `responseLatency`, `responseDuration`, and `runtimeOverhead`: + +![Trace details showing sub-span latency and runtime overhead timeline](./images/telemetry-traces-detail.png) + +## Related Resources + +- [Setup](../setup/) +- [Layer ARNs](../layer-arns/) +- [Overview](../) diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-log-explorer.png b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-log-explorer.png new file mode 100644 index 00000000..7c1fed92 Binary files /dev/null and b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-log-explorer.png differ diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-metrics-dashboard.png b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-metrics-dashboard.png new file mode 100644 index 00000000..b9296917 Binary files /dev/null and b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-metrics-dashboard.png differ diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-detail.png b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-detail.png new file mode 100644 index 00000000..e4299e32 Binary files /dev/null and b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-detail.png differ diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-list.png b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-list.png new file mode 100644 index 00000000..0019da00 Binary files /dev/null and b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/images/telemetry-traces-list.png differ diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/index.mdx b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/index.mdx new file mode 100644 index 00000000..4ed5ad10 --- /dev/null +++ b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/index.mdx @@ -0,0 +1,51 @@ +--- +title: "Lambda Telemetry Extension" +description: "Send AWS Lambda logs, metrics, and traces to KloudMate with a layer — no CloudFormation and no code changes." +sidebar: + label: "Overview" + order: 1 +--- +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The Lambda Telemetry Extension sends a function's logs, metrics, and traces to KloudMate. You add a layer to the function and set two environment variables — there's no code to change, no CloudFormation stack, and no CloudWatch subscription. + +## When to use it + +KloudMate gives you two ways to monitor Lambda, and they fit different needs. + +The [AWS account integration](../account-setup/) connects a whole account when you deploy one CloudFormation stack. From there KloudMate works automatically — it discovers your functions, reads their CloudWatch metrics, and subscribes their log groups for you, with no roles or subscriptions to wire up by hand. Use it to cover an account end to end without touching individual functions. + +The Telemetry Extension instruments the functions you choose. It adds traces on top of logs and metrics, and the data arrives within seconds instead of waiting on CloudWatch. Use it where you want richer, faster telemetry. + +| | AWS account integration | Telemetry Extension | +| --- | --- | --- | +| Setup | Deploy one CloudFormation stack | Add a layer and two env vars per function | +| Coverage | Every function in the account, automatically | The functions you add it to | +| Signals | Metrics and logs, from CloudWatch | Logs, metrics, **and traces** | +| Delivery | CloudWatch's delay (about a minute) | Within seconds | + +You can run both — the account integration for broad coverage, the extension where you want traces — with one overlap to watch for. + +:::caution +**Avoid duplicate logs.** The account integration subscribes every Lambda log group automatically. A function that also runs the extension sends its logs twice — once through CloudWatch, once through the extension. On those functions, set `OTEL_LOGS_EXPORTER=none` so the extension sends only metrics and traces, and CloudWatch keeps handling the logs. +::: + +## What you get + +- **Logs** — everything your function writes to stdout/stderr, searchable in Log Explorer. +- **Metrics** — invocation duration, billed duration, memory used, and cold-start time. +- **Traces** — one trace per invocation, linked to your AWS X-Ray trace when active tracing is on. + +## Get started + +<CardGrid> +<LinkCard title="Setup" href="./setup/" description="Add the layer and configure a function in a few minutes." /> +<LinkCard title="Layer ARNs" href="./layer-arns/" description="Find the layer ARN for your region and architecture." /> +<LinkCard title="Configuration & Data" href="./configuration/" description="Environment variables and the metrics you receive." /> +</CardGrid> + +## Related Resources + +- [AWS Integration](../) +- [AWS Lambda Monitoring](../lambda-monitoring/) +- [API Keys](../../platform/settings/api-keys/) diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/layer-arns.mdx b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/layer-arns.mdx new file mode 100644 index 00000000..4a70e684 --- /dev/null +++ b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/layer-arns.mdx @@ -0,0 +1,72 @@ +--- +title: "Lambda Layer ARNs" +description: "Per-region ARNs for the KloudMate Lambda Telemetry Extension layer, for both x86_64 and arm64 architectures." +sidebar: + label: "Layer ARNs" + order: 3 +--- + +Copy the layer ARN that matches your function's **Region** and **architecture**, then attach it in [Setup](../setup/). The layer is published publicly in every Region below, so any AWS account can use it. Copy the full ARN, including the version number at the end. + +:::caution +Two things have to match the function, or the layer won't attach (and the function won't start): + +- **Region** — use the ARN from the same Region as your function. +- **Architecture** — `lambda-telemetry-extension` for **x86_64**, `lambda-telemetry-extension-arm64` for **arm64**. +::: + +## x86_64 + +| Region | Layer ARN | +| --- | --- | +| ap-east-1 ⁺ | `arn:aws:lambda:ap-east-1:429321095007:layer:lambda-telemetry-extension:3` | +| ap-northeast-1 | `arn:aws:lambda:ap-northeast-1:429321095007:layer:lambda-telemetry-extension:4` | +| ap-northeast-2 | `arn:aws:lambda:ap-northeast-2:429321095007:layer:lambda-telemetry-extension:4` | +| ap-south-1 | `arn:aws:lambda:ap-south-1:429321095007:layer:lambda-telemetry-extension:4` | +| ap-southeast-1 | `arn:aws:lambda:ap-southeast-1:429321095007:layer:lambda-telemetry-extension:4` | +| ap-southeast-2 | `arn:aws:lambda:ap-southeast-2:429321095007:layer:lambda-telemetry-extension:4` | +| ca-central-1 | `arn:aws:lambda:ca-central-1:429321095007:layer:lambda-telemetry-extension:4` | +| eu-central-1 | `arn:aws:lambda:eu-central-1:429321095007:layer:lambda-telemetry-extension:4` | +| eu-north-1 | `arn:aws:lambda:eu-north-1:429321095007:layer:lambda-telemetry-extension:4` | +| eu-south-1 ⁺ | `arn:aws:lambda:eu-south-1:429321095007:layer:lambda-telemetry-extension:3` | +| eu-west-1 | `arn:aws:lambda:eu-west-1:429321095007:layer:lambda-telemetry-extension:4` | +| eu-west-2 | `arn:aws:lambda:eu-west-2:429321095007:layer:lambda-telemetry-extension:4` | +| eu-west-3 | `arn:aws:lambda:eu-west-3:429321095007:layer:lambda-telemetry-extension:4` | +| sa-east-1 | `arn:aws:lambda:sa-east-1:429321095007:layer:lambda-telemetry-extension:4` | +| us-east-1 | `arn:aws:lambda:us-east-1:429321095007:layer:lambda-telemetry-extension:4` | +| us-east-2 | `arn:aws:lambda:us-east-2:429321095007:layer:lambda-telemetry-extension:4` | +| us-west-1 | `arn:aws:lambda:us-west-1:429321095007:layer:lambda-telemetry-extension:4` | +| us-west-2 | `arn:aws:lambda:us-west-2:429321095007:layer:lambda-telemetry-extension:4` | + +## arm64 + +| Region | Layer ARN | +| --- | --- | +| ap-east-1 ⁺ | `arn:aws:lambda:ap-east-1:429321095007:layer:lambda-telemetry-extension-arm64:3` | +| ap-northeast-1 | `arn:aws:lambda:ap-northeast-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| ap-northeast-2 | `arn:aws:lambda:ap-northeast-2:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| ap-south-1 | `arn:aws:lambda:ap-south-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| ap-southeast-1 | `arn:aws:lambda:ap-southeast-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| ap-southeast-2 | `arn:aws:lambda:ap-southeast-2:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| ca-central-1 | `arn:aws:lambda:ca-central-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| eu-central-1 | `arn:aws:lambda:eu-central-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| eu-north-1 | `arn:aws:lambda:eu-north-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| eu-south-1 ⁺ | `arn:aws:lambda:eu-south-1:429321095007:layer:lambda-telemetry-extension-arm64:3` | +| eu-west-1 | `arn:aws:lambda:eu-west-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| eu-west-2 | `arn:aws:lambda:eu-west-2:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| eu-west-3 | `arn:aws:lambda:eu-west-3:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| sa-east-1 | `arn:aws:lambda:sa-east-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| us-east-1 | `arn:aws:lambda:us-east-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| us-east-2 | `arn:aws:lambda:us-east-2:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| us-west-1 | `arn:aws:lambda:us-west-1:429321095007:layer:lambda-telemetry-extension-arm64:4` | +| us-west-2 | `arn:aws:lambda:us-west-2:429321095007:layer:lambda-telemetry-extension-arm64:4` | + +:::note +⁺ `ap-east-1` (Hong Kong) and `eu-south-1` (Milan) are opt-in AWS Regions. The layer is available there once you've enabled the Region in your account. Need a Region that isn't listed? [Get in touch](../../../getting-started/get-help/) and we'll add it. +::: + +## Related Resources + +- [Setup](../setup/) +- [Configuration & Data](../configuration/) +- [Overview](../) diff --git a/src/content/docs/docs/aws-integration/lambda-telemetry-extension/setup.mdx b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/setup.mdx new file mode 100644 index 00000000..0dd0b0df --- /dev/null +++ b/src/content/docs/docs/aws-integration/lambda-telemetry-extension/setup.mdx @@ -0,0 +1,135 @@ +--- +title: "Set Up the Lambda Telemetry Extension" +description: "Attach the KloudMate layer to a Lambda function and point it at your OTLP endpoint with two environment variables." +sidebar: + label: "Setup" + order: 2 +--- +import { Steps, Tabs, TabItem } from '@astrojs/starlight/components'; + +You'll take one Lambda function from no telemetry to logs, metrics, and traces in KloudMate. It's about five minutes, and nothing in your function code changes. + +## Before you start + +- A KloudMate ingest API key. Create one under [API Keys](../../../platform/settings/api-keys/). +- Your KloudMate OTLP endpoint — `https://otel.kloudmate.com:4318` for most accounts. +- Permission to update the function's configuration (its layers and environment variables). + +## Add the layer and configure the function + +<Steps> + +1. **Find the layer ARN for your function.** It has to match the function's **Region** and **architecture** (x86_64 or arm64). Copy the matching ARN from [Layer ARNs](../layer-arns/). + +2. **Add the layer to the function.** A function can carry up to five layers, so this one sits alongside any you already use. + +3. **Set the environment variables:** + + ```bash + OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4318 + OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_KLOUDMATE_API_KEY + OTEL_SERVICE_NAME=my-service # optional; defaults to the function name + ``` + +4. **Deploy and invoke the function.** The change takes effect on the next invocation. Open the function in KloudMate to confirm data is arriving. + +</Steps> + +:::caution +If this account is already connected through the [AWS account integration](../../account-setup/), it subscribes every Lambda log group automatically — so this function's logs would arrive twice. Add `OTEL_LOGS_EXPORTER=none` to the variables above to send only metrics and traces from the extension, and let CloudWatch keep the logs. +::: + +## Configure it with infrastructure as code + +Drop the same layer ARN and environment variables into whatever you use to manage the function. The examples below use `us-east-1` (x86_64) — swap in the ARN for your Region and architecture, and your own API key. + +<Tabs> +<TabItem label="Terraform"> + +```hcl +resource "aws_lambda_function" "my_function" { + function_name = "my-service" + # ...your existing configuration... + + layers = [ + "arn:aws:lambda:us-east-1:429321095007:layer:lambda-telemetry-extension:4", + ] + + environment { + variables = { + OTEL_EXPORTER_OTLP_ENDPOINT = "https://otel.kloudmate.com:4318" + OTEL_EXPORTER_OTLP_HEADERS = "Authorization=YOUR_KLOUDMATE_API_KEY" + OTEL_SERVICE_NAME = "my-service" + } + } +} +``` + +</TabItem> +<TabItem label="AWS SAM"> + +```yaml +Resources: + MyFunction: + Type: AWS::Serverless::Function + Properties: + FunctionName: my-service + Layers: + - arn:aws:lambda:us-east-1:429321095007:layer:lambda-telemetry-extension:4 + Environment: + Variables: + OTEL_EXPORTER_OTLP_ENDPOINT: https://otel.kloudmate.com:4318 + OTEL_EXPORTER_OTLP_HEADERS: Authorization=YOUR_KLOUDMATE_API_KEY + OTEL_SERVICE_NAME: my-service +``` + +</TabItem> +<TabItem label="AWS CLI"> + +```bash +aws lambda update-function-configuration \ + --function-name my-service \ + --layers arn:aws:lambda:us-east-1:429321095007:layer:lambda-telemetry-extension:4 \ + --environment "Variables={ + OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.kloudmate.com:4318, + OTEL_EXPORTER_OTLP_HEADERS=Authorization=YOUR_KLOUDMATE_API_KEY, + OTEL_SERVICE_NAME=my-service + }" +``` + +</TabItem> +<TabItem label="Console"> + +1. Open the function in the AWS Lambda console. +2. Under **Layers**, choose **Add a layer → Specify an ARN**, paste the ARN for your Region and architecture, and add it. +3. Under **Configuration → Environment variables**, add `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, and (optionally) `OTEL_SERVICE_NAME`. +4. Save, then invoke the function. + +</TabItem> +</Tabs> + +## Confirm it's working + +Invoke the function once, then check KloudMate: + +- **Logs** — the function's output shows up in Log Explorer, tagged with its service name. +- **Metrics** — `faas.invoke_duration` and the related metrics appear for the function. +- **Traces** — with X-Ray active tracing enabled, each invocation produces a trace. + +Nothing within a minute? Set `OTEL_LOG_LEVEL=debug` and check the function's own CloudWatch logs — the extension reports what it's doing on lines tagged `[KloudMate]`. + +## Troubleshooting + +| What you see | Likely cause and fix | +| --- | --- | +| No data in KloudMate | Check that the API key in `OTEL_EXPORTER_OTLP_HEADERS` is valid and the endpoint is reachable. Turn on `OTEL_LOG_LEVEL=debug` and read the `[KloudMate]` lines. | +| `export rejected (401)` | The API key is missing or wrong. Recheck the `Authorization` value against [API Keys](../../../platform/settings/api-keys/). | +| Function fails to start | The layer's architecture doesn't match the function. Use the ARN that matches x86_64 or arm64. | +| Logs and metrics work, but no traces | Traces need X-Ray active tracing on the function. Turn it on, or expect logs and metrics only. | + +## Related Resources + +- [Layer ARNs](../layer-arns/) +- [Configuration & Data](../configuration/) +- [Overview](../) +- [API Keys](../../../platform/settings/api-keys/) diff --git a/src/content/docs/docs/azure-integration/manual-integration-of-azure-monitor.mdx b/src/content/docs/docs/azure-integration/manual-integration-of-azure-monitor.mdx index e7825772..c6e18035 100644 --- a/src/content/docs/docs/azure-integration/manual-integration-of-azure-monitor.mdx +++ b/src/content/docs/docs/azure-integration/manual-integration-of-azure-monitor.mdx @@ -4,7 +4,7 @@ description: "Documentation for Manual Integration of Azure Monitor" sidebar: order: 3 --- -This document provides a step-by-step manual approach to integrating Azure Monitor with KloudMate using **KloudMate agents**. The KloudMate agents allow you to scrape telemetry data from Azure Monitor resources and send them to KloudMate for centralized monitoring and analysis. +This document provides a step-by-step manual approach to integrating Azure Monitor with KloudMate using **KloudMate agents**. The KloudMate agent scrapes telemetry data from Azure Monitor resources and sends it to KloudMate for centralized monitoring and analysis. ## Pre-requisite 1. **Azure Service Principal for Authentication**: You will need a service principal with appropriate permissions to access Azure Monitor APIs. @@ -124,4 +124,4 @@ These commands will restart the KloudMate agent and display its current status. 2. In the Services window, locate the "`KloudMate agent`" service. 3. Right-click the service and select "`Restart`." -Subsequently, monitor the metrics on the KloudMate dashboard and set up an alarm to receive notifications if the potential metrics for a specific application rise. +Once data is flowing, monitor the metrics in KloudMate and set up an alert to be notified when metrics for a specific application exceed a threshold. diff --git a/src/content/docs/docs/getting-started/core-concepts.mdx b/src/content/docs/docs/getting-started/core-concepts.mdx index c39fb732..b7c1526b 100644 --- a/src/content/docs/docs/getting-started/core-concepts.mdx +++ b/src/content/docs/docs/getting-started/core-concepts.mdx @@ -5,7 +5,7 @@ sidebar: order: 2 --- -KloudMate relies on industry-standard observability frameworks to seamlessly collect and analyze data from your infrastructure and applications. Before you begin integrating your systems, it is helpful to understand three fundamental concepts: **Signals** , **OpenTelemetry** , and **eBPF**. +KloudMate relies on industry-standard observability frameworks to seamlessly collect and analyze data from your infrastructure and applications. Before integrating your systems, it helps to understand three fundamental concepts: **Signals** , **OpenTelemetry** , and **eBPF**. This 2-minute primer provides the essential context you need to get started. @@ -14,23 +14,27 @@ This 2-minute primer provides the essential context you need to get started. Observability is built upon three primary pillars of data, often referred to as "Signals". KloudMate correlates these signals to give you a unified view of your system's health. - **Logs:** Discrete, timestamped records of events that happened over time (e.g., an error message, a user login). They provide the granular context of *why* something happened. -- **Metrics:** Numerical representations of data measured over intervals of time (e.g., CPU usage, memory consumption, request latency). They are excellent for spotting trends and triggering alerts. +- **Metrics:** Numerical representations of data measured over intervals of time (e.g., CPU usage, memory consumption, request latency). They are useful for spotting trends and triggering alerts. - **Traces:** A representation of the end-to-end journey of a single request as it moves across distributed systems and microservices. They are vital for identifying performance bottlenecks and understanding *where* an issue occurred. ## 2. OpenTelemetry (OTel) [OpenTelemetry (OTel)](https://opentelemetry.io/) is an open-source, vendor-neutral standard for generating, collecting, and exporting observability signals (Logs, Metrics, and Traces). -KloudMate uses OTel as its primary data integration layer. This is a significant advantage for you: +KloudMate uses OTel as its primary data integration layer. This offers several advantages: - **No Vendor Lock-in:** You instrument your code once using OTel standards. You don't need proprietary KloudMate code in your applications. - **Standardized Data:** OTel ensures that the data sent to KloudMate is uniformly structured, making correlation and analysis much more powerful. - **The Collector:** OTel provides a "Collector" component that can receive, process, and export data. When you set up KloudMate integrations, you are often configuring an OTel Collector to forward data to the KloudMate platform. *For more details, see our dedicated [OpenTelemetry Guide](../../opentelemetry/).* -## 3. eBPF (Extended Berkeley Packet Filter) +## 3. Service-Level Objectives (SLOs) -[eBPF](https://ebpf.io/) is a revolutionary technology that allows programs to run directly within the operating system kernel safely and efficiently, without requiring kernel source code changes or loading kernel modules. +A **service-level objective (SLO)** turns the question *"is this service reliable?"* into a number — a target percentage measured over a time window. KloudMate lets you anchor an SLO on a service, pick the signal you want to measure (the SLI — incident availability, span latency, error rate, log severity, custom metric, and more), and watch compliance trend over time. **Burn-rate alerts** page on dangerously fast error-budget consumption before the budget runs out. See [Reliability](../../reliability/) for the full surface. + +## 4. eBPF (Extended Berkeley Packet Filter) + +[eBPF](https://ebpf.io/) is a technology that allows programs to run directly within the operating system kernel safely and efficiently, without requiring kernel source code changes or loading kernel modules. In the context of observability and the KloudMate Agent (KloudMate Agent): - **Zero-Code Instrumentation:** eBPF allows the agent to observe network traffic, application performance, and system calls automatically. You do not need to modify your application code to get deep insights. @@ -39,7 +43,7 @@ In the context of observability and the KloudMate Agent (KloudMate Agent): ## Next Steps -Now that you understand the core concepts of how data is generated (Signals), standardized (OpenTelemetry), and efficiently collected (eBPF), you are ready to start sending data to KloudMate. +Now that you understand how data is generated (Signals), standardized (OpenTelemetry), and collected (eBPF), you are ready to start sending data to KloudMate. - [Setting up KloudMate](../setting-up-kloudmate/) - [Learn about the KloudMate Agent](../../kloudmate-agent/) diff --git a/src/content/docs/docs/getting-started/get-help.mdx b/src/content/docs/docs/getting-started/get-help.mdx index 9ef4db8f..028ef060 100644 --- a/src/content/docs/docs/getting-started/get-help.mdx +++ b/src/content/docs/docs/getting-started/get-help.mdx @@ -4,11 +4,11 @@ description: "Find support channels, community links, and ways to contact the Kl sidebar: order: 5 --- -**Questions, feedback, bug reports or, feature requests; we are always listening.** Our growing community and support teams are available **round-the-clock**, to help you with anything and everything. You can reach our support teams by: +**For questions, feedback, bug reports, or feature requests, we are always available.** Our growing community and support teams are available **round-the-clock**, to help you with anything and everything. You can reach our support teams by: -- Using the '**live chat** ' widget, available in the bottom right corner of [the website](https://www.kloudmate.com) +- Using the live chat widget in the bottom right corner of [the website](https://www.kloudmate.com) - Writing to us at [hello@kloudmate.com](mailto\:hello@kloudmate.com) -We are constantly responding to queries via email. We are also building an awesome [Slack Community](https://join.slack.com/t/kloudmate/shared_invite/zt-zvzq985i-Khwnq2jsm5uLTdBe87gCgw) of developers and customers, who interact, share their experiences and support each other. The community offers faster responses to issues or queries that you might have. +We respond to queries via email and have a [Slack Community](https://join.slack.com/t/kloudmate/shared_invite/zt-zvzq985i-Khwnq2jsm5uLTdBe87gCgw) of developers and customers who interact, share experiences, and support each other. The community offers faster responses to issues or queries that you might have. [Join us on Slack](https://join.slack.com/t/kloudmate/shared_invite/zt-zvzq985i-Khwnq2jsm5uLTdBe87gCgw) diff --git a/src/content/docs/docs/getting-started/sending-data-to-kloudmate.mdx b/src/content/docs/docs/getting-started/sending-data-to-kloudmate.mdx index 4cfd9218..d206233d 100644 --- a/src/content/docs/docs/getting-started/sending-data-to-kloudmate.mdx +++ b/src/content/docs/docs/getting-started/sending-data-to-kloudmate.mdx @@ -63,7 +63,7 @@ If you are unsure where to begin, use this table to map your immediate goal to t ## Common Rollout Pattern -For teams starting from scratch, we recommend this phased approach: +For teams starting from scratch, the following phased approach is recommended: 1. **Infrastructure First:** Start with the [KloudMate Agent](../../kloudmate-agent/) to get baseline telemetry, host metrics, and zero-code visibility. 2. **Application Tracing:** Add [OpenTelemetry](../../opentelemetry/) instrumentation to your applications where you need richer distributed tracing or custom metrics. diff --git a/src/content/docs/docs/getting-started/setting-up-kloudmate.mdx b/src/content/docs/docs/getting-started/setting-up-kloudmate.mdx index dba538b1..c49838f8 100644 --- a/src/content/docs/docs/getting-started/setting-up-kloudmate.mdx +++ b/src/content/docs/docs/getting-started/setting-up-kloudmate.mdx @@ -1,52 +1,107 @@ --- title: "Setting Up KloudMate" -description: "Create a workspace and complete the initial KloudMate onboarding flow." +description: "Sign up, walk through the onboarding wizard, and pick the right ingestion path for what you actually want to monitor." sidebar: order: 3 --- -Welcome to KloudMate! Setting up your account takes just a few minutes. This guide will walk you through creating your organization, setting up your first workspace, and preparing to ingest telemetry data. +import { LinkCard, CardGrid, Steps } from '@astrojs/starlight/components'; -## Signing Up +This page is your starting point. You'll sign up, run through onboarding, send your first telemetry event, and determine what to set up next — based on what you want to monitor: infrastructure, applications, databases, cloud services, or something else. -1. [Click here](https://app.kloudmate.com/signup) to go to the KloudMate signup page. -2. Fill in the signup form with your details (work email only). +Already have an account and just want to send data? Jump to [Sending Data to KloudMate](../sending-data-to-kloudmate/). -![image](../images/setting-up-kloudmate-1.png) +## 1. Sign up -3. Check your email for a verification code from `noreply@kloudmate.com` (Don't forget the spam folder!). -4. Enter the verification code on the KloudMate window and click **Next**. -5. Enter your organization name and workspace name. Click **Next**. +<Steps> +1. Open the [KloudMate signup page](https://app.kloudmate.com/signup). +2. Fill in the form with a work email address. -![image](../images/setting-up-kloudmate-2.png) + ![Signup form](../images/setting-up-kloudmate-1.png) -![image](../images/setting-up-kloudmate-3.png) +3. Check your inbox for a verification code from `noreply@kloudmate.com`. Check your spam folder if it doesn't arrive. +4. Enter the code on the next screen and click **Next**. -## Selecting a Data Source + ![Verification code](../images/setting-up-kloudmate-2.png) +</Steps> -After creating your workspace, you will be prompted to select a method to send data to KloudMate. +## 2. Walk through the onboarding wizard -![image](../images/setting-up-kloudmate-4.png) +Onboarding is a five-step wizard. Most steps take seconds; the only one with real work is the data-source step. -### Using OpenTelemetry +### Step 1 — Create your organization -- Select **OpenTelemetry** if you want to generate, collect, and export metrics, logs, and traces to KloudMate using the OpenTelemetry SDKs or Collector. -- The **Endpoint** and **API Key** required to send data to KloudMate are already available on this page. Use these details while configuring your OpenTelemetry instrumentation or Collector. -- If you prefer a simpler setup, you can install the **KloudMate Agent** by clicking **Install Agent** above. The agent is OpenTelemetry-based and comes preconfigured to send telemetry data to KloudMate. +Pick a name for your organization. Your browser timezone is captured as the org-wide default for alert schedules and report cadences — change it later under **Settings → Organization**. -![image](../images/setting-up-kloudmate-5.png) +### Step 2 — Tell us about your stack -- For detailed steps on installing and configuring agents, refer to the [KloudMate Agent](../../kloudmate-agent/) documentation. -- For a complete guide to sending data with OpenTelemetry, see the [OpenTelemetry guide](../../opentelemetry/). +Three quick picks that tailor the next step to your setup: -### Using AWS Integration +- **Your role** — Developer, SRE, Platform, or Eng. Manager. +- **Primary language** — Node, Python, Java, Go, .NET, Ruby, PHP, or Other. +- **Where you run your services** — Kubernetes, VMs, Serverless (AWS), or Mixed. -Choose **AWS Integration** if you want to connect an AWS account through the built-in onboarding flow. +These drive the install-method recommendation in Step 4 and pre-tune the snippets you'll see. Change any of them later under **Settings**. -For the full setup steps, see [AWS Account Setup](../../aws-integration/account-setup/). +### Step 3 — Name your workspace -## Related Resources +Workspaces separate environments, teams, or products inside one organization — for example, `prod`, `staging`, and `dev` as three workspaces sharing the same org. Pick a name; add more workspaces later under **Settings → Workspaces**. -- [AWS Account Setup](../../aws-integration/account-setup/) -- [Sending Data to KloudMate](../sending-data-to-kloudmate/) -- [Get Help](../get-help/) +### Step 4 — Send your first event + +Two options appear, with one marked **Recommended for your stack** based on the deployment you picked in Step 2: + +| If you picked… | Recommended path | +|---|---| +| **Kubernetes**, **VMs**, or **Mixed** | **OpenTelemetry** — install the KloudMate agent (Helm chart for K8s, Linux/Windows binary for VMs). Auto-instruments Node, Python, Java, .NET, and Go pods on K8s with no code changes. | +| **Serverless (AWS)** | **AWS Integration** — connect your AWS account; Lambda, EC2, RDS metrics and CloudWatch logs flow in without an agent install. | + +Select the recommended option unless your setup requires otherwise. The other option is still available — connect AWS even from a Kubernetes deployment if you want both. The next section covers less common cases. + +The OpenTelemetry path opens install instructions in a new tab and advances the wizard to Step 5. The AWS path runs the AWS onboarding flow inline. + +### Step 5 — Verify ingestion + +The verify step waits for any signal — trace, log, metric, RUM event — to land in the workspace. As soon as the first one arrives, the step ticks complete and the wizard finishes. To continue later, click **Skip for now** — the home-page checklist picks up where you left off. + +## 3. Pick the right ingestion path for your goal + +The Step 4 recommendation is a good default, but the right path depends on what you're actually monitoring. Use this table to match your goal to the page that has the install instructions. + +| What you want to monitor | Best path | Notes | +|---|---|---| +| **Hosts, VMs, or Kubernetes infra** (CPU, memory, disk, network, pod health) | [KloudMate Agent](../../kloudmate-agent/) | One agent install. eBPF-powered, low overhead, zero code changes. | +| **Application traces and latency** (APM) | [OpenTelemetry SDK](../../opentelemetry/) — or the KloudMate Agent on K8s for auto-instrumentation | On Kubernetes the agent auto-instruments Node, Python, Java, .NET, and Go pods. Outside K8s, install the OTel SDK directly in your app. | +| **Application logs** | [KloudMate Agent](../../kloudmate-agent/) — or push directly via [OpenTelemetry](../../opentelemetry/) | The agent tails common log paths automatically. The OTel logs SDK gives you finer control over structured logging. | +| **Databases** (Postgres, MySQL, Mongo, Redis, etc.) | [Database Monitoring](../../database-monitoring/) | Connect the database; query stats, slow logs, and connection health stream in. | +| **AWS services** (Lambda, EC2, RDS, ECS, DynamoDB, …) | [AWS Integration](../../aws-integration/) | Agentless cloud-platform pull via the AWS API. | +| **Azure services** | [Azure Integration](../../azure-integration/) | Azure Monitor + Event Hub. | +| **Existing Prometheus metrics** | [Prometheus Integration](../../infrastructure/integrations/prometheus-integration/) | Forward your Prometheus metrics via the OTel-based remote-write path. | +| **Existing OpenTelemetry pipelines** | [OpenTelemetry Collector](../../opentelemetry/) | Add an OTLP exporter — no changes to your collector-first architecture. | +| **Synthetic monitoring** (uptime, browser checks) | [Synthetics](../../synthetic/) | Set up checks from the UI; no agent or SDK required. | +| **Real-User Monitoring (RUM)** | [RUM](../../rum/) | Drop the browser SDK into your frontend to capture session, performance, and error data. | + +For a side-by-side decision walkthrough and a phased rollout pattern, see [Sending Data to KloudMate](../sending-data-to-kloudmate/). + +## 4. What to do next + +When the wizard finishes, you land on the home page with a **Setup checklist** drawer open. It lists the next steps worth completing — instrument an application, build a dashboard, create an alert, wire a notification channel, invite a teammate. Each row jumps you straight to the right page, with the option to ask the assistant if you'd rather be walked through. + +Re-open it any time from the **Setup** pill in the header. + +## Where to go from here + +<CardGrid> + <LinkCard title="Sending Data to KloudMate" description="Detailed comparison of every ingestion path and a phased rollout pattern." href="../sending-data-to-kloudmate/" /> + <LinkCard title="KloudMate Agent" description="Install the agent for infrastructure, logs, and zero-code APM via eBPF." href="../../kloudmate-agent/" /> + <LinkCard title="OpenTelemetry guide" description="Use the OTel SDK or Collector to ship application telemetry." href="../../opentelemetry/" /> + <LinkCard title="AWS Integration" description="Connect an AWS account for Lambda, EC2, RDS, and more." href="../../aws-integration/" /> + <LinkCard title="Create your first alert" description="Watch a metric, define a threshold, route notifications." href="../../alerts/create-alerts/" /> + <LinkCard title="Reliability & SLOs" description="Set service-level objectives and get paged on burn-rate spikes." href="../../reliability/" /> +</CardGrid> + +## Related resources + +- [What is KloudMate?](../what-is-kloudmate/) — concepts and platform overview. +- [Core concepts](../core-concepts/) — signals, OpenTelemetry, eBPF, SLOs. +- [Get help](../get-help/) — support and community. diff --git a/src/content/docs/docs/getting-started/what-is-kloudmate.mdx b/src/content/docs/docs/getting-started/what-is-kloudmate.mdx index abcec5ff..e221f526 100644 --- a/src/content/docs/docs/getting-started/what-is-kloudmate.mdx +++ b/src/content/docs/docs/getting-started/what-is-kloudmate.mdx @@ -4,9 +4,9 @@ description: "Learn what KloudMate is and how it brings logs, metrics, traces, a sidebar: order: 1 --- -[KloudMate](https://www.kloudmate.com) is a comprehensive **observability platform** that assists developers in identifying & troubleshooting issues and optimizing application performance & health across distributed environments. +[KloudMate](https://www.kloudmate.com) is a comprehensive **observability platform** that helps developers identify and troubleshoot issues and optimize application performance across distributed environments. -KloudMate offers **end-to-end visibility** into application infrastructures, distributed systems, and development environments. It comes packed with **OpenTelemetry integration,** a real-time alerting system, a user-friendly interface, powerful dashboards, and many more state-of-the-art features to provide reinforced observability and monitoring. **KloudMate offers a Single-Source-of-Truth approach** for observability by enabling developers to gather and consolidate telemetry data from multiple distributed systems using **OpenTelemetry.** With KloudMate, you can get a detailed diagnosis of your application infrastructure, pinpoint & troubleshoot errors, drill down to associated logs, metrics, or traces, and fix issues **in a matter of minutes**. +KloudMate offers **end-to-end visibility** into application infrastructures, distributed systems, and development environments. It includes **OpenTelemetry integration,** a real-time alerting system, powerful dashboards, and more — all designed to provide comprehensive observability and monitoring. **KloudMate offers a Single-Source-of-Truth approach** for observability by enabling developers to gather and consolidate telemetry data from multiple distributed systems using **OpenTelemetry.** With KloudMate, you can diagnose your application infrastructure, pinpoint and troubleshoot errors, and drill down to associated logs, metrics, or traces. And to keep things neat and organized, KloudMate lets you create multiple **Workspaces**. You can use workspaces to group multiple teams, accounts, or orgs. For example, you can create separate workspaces for development, production, etc. @@ -16,9 +16,9 @@ Observability is the practice of using tools to aggregate, correlate, and analyz Observability essentially depends upon the ability to log, monitor, and trace different servers, applications, systems, and data processes across the lifecycle of the environment. -Observability on KloudMate is a revolutionary new way to gain deep visibility into your distributed application resources, attain faster insights, or automate problem identification & resolution. +KloudMate's observability tools give you visibility into distributed application resources, faster insights, and automated problem identification and resolution. -The article(s) listed in this section of the documentation will help you get started with KloudMate. +The pages in this section will help you get started with KloudMate. Related Resources diff --git a/src/content/docs/docs/images/issues/issues-1.jpeg b/src/content/docs/docs/images/issues/issues-1.jpeg deleted file mode 100644 index b768b0ac..00000000 Binary files a/src/content/docs/docs/images/issues/issues-1.jpeg and /dev/null differ diff --git a/src/content/docs/docs/images/issues/issues-1.png b/src/content/docs/docs/images/issues/issues-1.png deleted file mode 100644 index 4f1a4821..00000000 Binary files a/src/content/docs/docs/images/issues/issues-1.png and /dev/null differ diff --git a/src/content/docs/docs/images/issues/issues-2.png b/src/content/docs/docs/images/issues/issues-2.png deleted file mode 100644 index 296bdcb4..00000000 Binary files a/src/content/docs/docs/images/issues/issues-2.png and /dev/null differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/_meta.json b/src/content/docs/docs/incident-management/escalation-policy/_meta.json index 3bc64b7e..31596c27 100644 --- a/src/content/docs/docs/incident-management/escalation-policy/_meta.json +++ b/src/content/docs/docs/incident-management/escalation-policy/_meta.json @@ -1,4 +1,4 @@ { "label": "Escalation Policies", - "order": 6 + "order": 7 } diff --git a/src/content/docs/docs/incident-management/escalation-policy/adding-escalation-policy.mdx b/src/content/docs/docs/incident-management/escalation-policy/adding-escalation-policy.mdx index 6198a03b..18b8a4f2 100644 --- a/src/content/docs/docs/incident-management/escalation-policy/adding-escalation-policy.mdx +++ b/src/content/docs/docs/incident-management/escalation-policy/adding-escalation-policy.mdx @@ -1,39 +1,69 @@ --- title: "Create an Escalation Policy" -description: "Create an escalation policy with one or more notification and timeout steps." +description: "Build an escalation policy: set the initial wait and repeat, then add steps that page users, on-call schedules, and Slack channels." sidebar: label: "Create an Escalation Policy" order: 2 --- -Use the steps below to create a new escalation policy. +import { Steps } from '@astrojs/starlight/components'; -1. Click **Add Escalation Policy** from the escalation policies page. +This page walks through building a policy from scratch — the policy-level settings first, then the steps that do the paging. -![image](./images/adding-escalation-policy-1.jpeg) +## Create the policy -2. Enter a **Name** and **Description**. -3. Set the initial **Wait Time** in minutes. +<Steps> +1. Open **Incident Management → Escalation Policies** and click **Add Escalation Policy**. -![image](./images/adding-escalation-policy-1.png) +2. Enter a **name** and **description**. -4. Under **Add Escalation Steps**, configure the notification targets for the first step: - - Select the notification channel - - Select the assignee - - Use **Add Another Alert** to add multiple channels in the same step +3. Set the **wait time** — minutes before the first step fires. Leave it at `0` to page immediately. -:::note - If you plan to notify Slack, first complete [Slack Integration](../../slack-integration/). +4. Set the **repeat count** — how many extra times to run the whole policy from step 1 if nobody acknowledges. `0` runs the policy once. +</Steps> + +![Escalation policy form](./images/escalation-policy-form.png) + +## Add steps + +Each step names its recipients and how long to wait before escalating. + +<Steps> +1. In a step, add one or more **recipients**: + - **Users** — pick a user, then choose how to notify them: **email**, **SMS**, or **voice**. Select at least one method. SMS and voice are disabled (with a tooltip) unless your plan includes them and the user has a verified phone number. + - **On-call schedule** — pick a schedule; the step notifies whoever is on call when it fires — possibly several people. Requires the Pro plan. + - **Slack channel** — pick a connected channel and set its fallback order (`0` is primary). Connect Slack first under [Slack Integration](../../slack-integration/). + +2. Set **Escalate after** — minutes to wait for an acknowledgement before the next step fires. + +3. *(Optional)* Turn on **retries** for the step: set a **retry interval** and a **max retries** count to re-page the same recipients until someone acknowledges. + +4. Add more steps as needed. The connector between steps spells out what happens next — "Escalate after 10 minutes • then escalate to step 2", or "then repeat the policy from step 1", or "then stop". + +5. Click **Save**. +</Steps> + +![Escalation step with mixed recipients](./images/escalation-step-recipients.png) + +:::caution[Retry interval must be shorter than escalate-after] +Retries only run during a step's escalate-after window. If the retry interval is at least as long as that window, the first retry never fires before the policy advances — the editor flags this with a warning. Set the retry interval shorter than escalate-after. ::: -5. Set the **Escalate After** timeout in minutes for that step. +![Step retry configuration and connector](./images/escalation-step-retry.png) + +## A worked example -![image](./images/adding-escalation-policy-2.png) +A policy that pages on-call first, then widens if no one answers: -6. Click **Add Escalation** to create additional steps as needed. +1. **Wait 0 min** — step 1 fires as soon as the incident opens. +2. **Step 1** pages the **on-call schedule** plus a **backup user** by email and SMS. **Escalate after 5 min.** +3. No ack in 5 minutes → **Step 2** posts to a **Slack channel** and pages the **manager** by voice. **Escalate after 10 min.** +4. Still no ack → with **repeat count 1**, the policy runs from step 1 once more. -![image](./images/adding-escalation-policy-3.png) +As soon as anyone acknowledges, the policy stops. -7. Click **Save** when the policy is complete. +## Related -![image](./images/adding-escalation-policy-4.png) +- [Escalation Policies overview](../) — the model and timing. +- [On-Call Schedules](../../oncall/) — what a schedule recipient resolves to. +- [Slack Integration](../../slack-integration/) — required for channel recipients. diff --git a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-1.jpeg b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-1.jpeg index 6c9c438c..cda7d54d 100644 Binary files a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-1.jpeg and b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-1.jpeg differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-detail.png b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-detail.png new file mode 100644 index 00000000..c282de40 Binary files /dev/null and b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-detail.png differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-form.png b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-form.png new file mode 100644 index 00000000..ef9db0ec Binary files /dev/null and b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-policy-form.png differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-recipients.png b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-recipients.png new file mode 100644 index 00000000..053565a7 Binary files /dev/null and b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-recipients.png differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-retry.png b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-retry.png new file mode 100644 index 00000000..4ef91f87 Binary files /dev/null and b/src/content/docs/docs/incident-management/escalation-policy/images/escalation-step-retry.png differ diff --git a/src/content/docs/docs/incident-management/escalation-policy/index.mdx b/src/content/docs/docs/incident-management/escalation-policy/index.mdx index 2d70bee0..63d1bb7e 100644 --- a/src/content/docs/docs/incident-management/escalation-policy/index.mdx +++ b/src/content/docs/docs/incident-management/escalation-policy/index.mdx @@ -1,35 +1,73 @@ --- title: "Escalation Policies" -description: "Define who gets notified, how they are notified, and when incidents escalate." +description: "Define who gets notified, how, and when an unacknowledged incident escalates to the next step." sidebar: label: "Overview" order: 1 --- -Escalation policies control how incident notifications progress when an incident remains unacknowledged. +An escalation policy decides who gets paged when an incident opens, and what happens if no one acknowledges. A service points at one policy as its default; when an incident opens, the policy runs until someone acknowledges, the steps run out, or the policy finishes repeating. -They let you define: +![image](./images/escalation-policy-1.jpeg) -- Who should be notified -- Which notification channel should be used -- How long each escalation step should wait before moving to the next step +The policies list shows each policy's name, description, initial wait time, and number of steps. -![image](./images/escalation-policy-1.jpeg) +## How a policy is built + +A policy has two levels: the policy itself, and an ordered list of steps. + +**Policy** + +- **Name** and **description**. +- **Wait time** — minutes to wait before the first step fires. `0` pages immediately. +- **Repeat count** — how many extra times to run the whole policy from step 1 if nobody acknowledges. `0` means run once; `2` means three passes in total. + +**Step** + +- **Recipients** — one or more targets (see below). +- **Escalate after** — minutes to wait for an acknowledgement before the next step fires. +- **Retries** *(optional)* — re-page this step's recipients on a timer until someone acknowledges (covered below). + +## Step recipients + +A single step can mix three kinds of recipient: -The escalation policies list shows each policy’s name, description, wait time, and number of escalation steps. +- **Users** — each notified by **email**, **SMS**, or **voice**. Pick at least one method per user. SMS and voice need a paid plan and a verified phone number; the editor disables them with a tooltip when either is missing. +- **On-call schedules** — the step notifies whoever is on call on the schedule at the moment it fires. A schedule can resolve to several people at once, and each is notified through their own methods. On-call schedules need the Pro plan. +- **Slack channels** — posted to a shared channel. A channel's **fallback order** lets you tier channels: `0` is primary, higher numbers are fallbacks. Per-user Slack DMs aren't supported — only shared channels. :::note - A single escalation step can notify multiple targets through multiple channels. +A single step can notify several users, an on-call schedule, and a Slack channel at once. ::: -## Policy Details +## How the timing plays out -Open any escalation policy to review: +Picture a two-step policy with a repeat: + +1. **Wait 0 min** — the incident opens and step 1 fires immediately. +2. **Step 1** pages the on-call schedule and a backup user. **Escalate after 5 min.** +3. No acknowledgement in 5 minutes → **Step 2** pages a Slack channel and the team manager. **Escalate after 10 min.** +4. Still no acknowledgement → with **repeat count 1**, the policy loops back to step 1 and runs the whole thing once more. + +The moment someone acknowledges, the policy stops — no more steps, no more repeats. + +## Re-page a step until someone acknowledges + +Within a step you can re-notify the same recipients on a timer instead of waiting for the next step. Set: + +- **Retry interval** (ack timeout) — minutes between re-pages. +- **Max retries** — how many times to re-page. + +Retries run only during that step's **escalate-after** window. Once the policy advances, any remaining retries are skipped. + +:::caution[Keep the retry interval shorter than the step's wait] +If the retry interval is at least as long as the step's escalate-after time, the first retry may never run before the policy moves on. The editor warns you when that happens — set the retry interval shorter than escalate-after so retries actually fire. +::: -- Initial wait time -- Notification targets for each step -- Escalation timeout for each step +![Escalation policy detail with steps](./images/escalation-policy-detail.png) -![image](./images/escalation-policy-1.png) +## Next -Use the settings menu to edit or archive the policy. +- [Create an escalation policy](./adding-escalation-policy/) — the step-by-step build. +- [On-Call Schedules](../oncall/) — what a step pages when it targets a schedule. +- [Slack Integration](../slack-integration/) — connect Slack before adding channel recipients. diff --git a/src/content/docs/docs/incident-management/images/incident-lifecycle-pipeline.png b/src/content/docs/docs/incident-management/images/incident-lifecycle-pipeline.png new file mode 100644 index 00000000..4c413e27 Binary files /dev/null and b/src/content/docs/docs/incident-management/images/incident-lifecycle-pipeline.png differ diff --git a/src/content/docs/docs/incident-management/images/slack-integration-1.png b/src/content/docs/docs/incident-management/images/slack-integration-1.png index 5f26f178..b8f622d6 100644 Binary files a/src/content/docs/docs/incident-management/images/slack-integration-1.png and b/src/content/docs/docs/incident-management/images/slack-integration-1.png differ diff --git a/src/content/docs/docs/incident-management/images/what-is-incident-management_-1.jpeg b/src/content/docs/docs/incident-management/images/what-is-incident-management_-1.jpeg index ccf6885e..299f36ff 100644 Binary files a/src/content/docs/docs/incident-management/images/what-is-incident-management_-1.jpeg and b/src/content/docs/docs/incident-management/images/what-is-incident-management_-1.jpeg differ diff --git a/src/content/docs/docs/incident-management/incident-lifecycle.mdx b/src/content/docs/docs/incident-management/incident-lifecycle.mdx index 44c41cda..fcf8228e 100644 --- a/src/content/docs/docs/incident-management/incident-lifecycle.mdx +++ b/src/content/docs/docs/incident-management/incident-lifecycle.mdx @@ -1,58 +1,72 @@ --- title: "Incident Lifecycle" -description: "Understand how incidents move from alert trigger to acknowledgment, escalation, and resolution." +description: "Understand how incidents move from alert trigger through routing, on-call paging, escalation, and resolution." sidebar: label: "Incident Lifecycle" order: 3 --- -An incident lifecycle begins when an alert or event triggers an incident and ends when that incident is resolved. +An incident's life begins when an alert triggers it and ends when someone resolves it. In between, KloudMate moves it through a coordinated flow that pulls in alert sources, routing rules, services, escalation policies, and on-call schedules. -In KloudMate, incidents move through a coordinated flow that combines alerts, integrations, services, and escalation policies. +![Incident lifecycle pipeline](./images/incident-lifecycle-pipeline.png) -![image](./images/incident-lifecycle-1.jpg) +## Alert -## Alerts +An alert fires when a monitored condition or threshold is met. It's the starting signal for an incident. -Alerts are generated when monitored conditions or thresholds are met. They act as the starting signal for incident creation. +## Alert source -## Integrations +An [alert source](../integrations/) receives alerts from an external system — KloudMate alerts, CloudWatch, or a generic webhook — and maps each payload into incident fields. Every alert source belongs to a **service** and can set a default **escalation policy**. -Integrations receive alerts from external systems and route them into KloudMate Incident Management. +## Routing rule -Important rules: +Before an incident opens, the alert runs through your [routing rules](../routing-rules/). A matching rule decides: -- Each integration belongs to a service -- Each integration can have one escalation policy -- If an integration does not define its own escalation policy, it inherits the default policy from its service +- which **service** the incident opens against, +- which **escalation policy** pages, +- the **severity**, +- whether to **suppress** the alert entirely, and +- whether to **add responders** or **notify on-call schedules**. -## Incidents +This is where raw alert traffic becomes the right incident — or gets dropped before it pages anyone. -An incident is created when an integrated alert is triggered and requires operational attention. +### Where routing rules fit -## Services +Routing rules sit between the alert source and the incident. If no rule matches, the incident inherits its service and escalation policy from the alert source and service defaults, so routing rules are an optional layer of control rather than a requirement. -Services group related incidents together by microservice, application area, or ownership boundary. +## Incident -Each service: +Unless a rule suppresses it, an incident is created and needs operational attention. It records its service, source, severity, and a full activity timeline. See [Incidents](../incidents/). -- Has a default escalation policy -- Can contain multiple integrations -- Can allow an integration to override the default escalation policy +## Service -## Escalation Policies +[Services](../services/) group related incidents by microservice, application area, or ownership. Each service has a default escalation policy and can own multiple alert sources. -An escalation policy defines who should be notified and how the incident response should progress if the incident is not acknowledged in time. +## Escalation policy -## Escalation Steps +An [escalation policy](../escalation-policy/) defines who is notified and how the response progresses if no one acknowledges in time. A policy waits an initial delay, then works through its steps; if nobody acknowledges, it can repeat from the top. -Escalation steps form the escalation chain inside a policy. +## Escalation steps -When an incident reaches a responder set: +Each step notifies one or more recipients — users, on-call schedules, or Slack channels: -- If the incident is acknowledged or resolved before the escalation timeout, the workflow stops at that step -- If the incident is not acknowledged in time, the next escalation step is triggered +- If the incident is acknowledged or resolved before the step's timer elapses, the policy stops there. +- If not, the next step fires. + +## On-call resolution + +When a step targets an [on-call schedule](../oncall/) instead of a named person, KloudMate resolves the schedule at that exact moment — applying any override, otherwise taking the union of its active rotations — and notifies whoever is on call right then, which may be several people. + +### Where on-call fits + +On-call schedules don't notify anyone on their own. They're a target an escalation step (or a routing rule) points at, resolved to whoever is on call only when the notification needs to go out. That late resolution is why a last-minute override or rotation change still takes effect. + +## Resolution + +The incident is resolved when a responder marks it resolved — directly, or through Slack ChatOps. While you respond, you can [publish the incident to a status page](../status-pages/posting-incidents/) to keep customers informed. ## Related Resources - [What Is Incident Management?](../what-is-incident-management/) +- [Routing Rules](../routing-rules/) - [Escalation Policies](../escalation-policy/) +- [On-Call Schedules](../oncall/) diff --git a/src/content/docs/docs/incident-management/incidents/images/incident-activity-timeline.png b/src/content/docs/docs/incident-management/incidents/images/incident-activity-timeline.png new file mode 100644 index 00000000..194c4fb7 Binary files /dev/null and b/src/content/docs/docs/incident-management/incidents/images/incident-activity-timeline.png differ diff --git a/src/content/docs/docs/incident-management/incidents/images/incident-details-1.png b/src/content/docs/docs/incident-management/incidents/images/incident-details-1.png index 3ca881af..6bf0da41 100644 Binary files a/src/content/docs/docs/incident-management/incidents/images/incident-details-1.png and b/src/content/docs/docs/incident-management/incidents/images/incident-details-1.png differ diff --git a/src/content/docs/docs/incident-management/incidents/images/incident-publish-status-page.png b/src/content/docs/docs/incident-management/incidents/images/incident-publish-status-page.png new file mode 100644 index 00000000..60ab17cd Binary files /dev/null and b/src/content/docs/docs/incident-management/incidents/images/incident-publish-status-page.png differ diff --git a/src/content/docs/docs/incident-management/incidents/images/incident-responders.png b/src/content/docs/docs/incident-management/incidents/images/incident-responders.png new file mode 100644 index 00000000..d0052663 Binary files /dev/null and b/src/content/docs/docs/incident-management/incidents/images/incident-responders.png differ diff --git a/src/content/docs/docs/incident-management/incidents/images/incidents-list-filters.png b/src/content/docs/docs/incident-management/incidents/images/incidents-list-filters.png new file mode 100644 index 00000000..0302597b Binary files /dev/null and b/src/content/docs/docs/incident-management/incidents/images/incidents-list-filters.png differ diff --git a/src/content/docs/docs/incident-management/incidents/incident-details.mdx b/src/content/docs/docs/incident-management/incidents/incident-details.mdx index abbab597..10f3213d 100644 --- a/src/content/docs/docs/incident-management/incidents/incident-details.mdx +++ b/src/content/docs/docs/incident-management/incidents/incident-details.mdx @@ -1,48 +1,68 @@ --- title: "Incident Details" -description: "Review responders, status, metrics, notes, logs, and alarms for an incident." +description: "Read the activity timeline, manage responders and severity, and publish an incident to a status page." sidebar: label: "Incident Details" order: 3 --- -Open any incident from the incident list to view its detail page. +Open any incident from the list to view its detail page. It's the single place to see what happened, who was paged, and how the response unfolded. ![image](./images/incident-details-1.png) -The header shows: +## The header -- Assigned responders -- Related service -- Source integration -- Applied escalation policy +The header identifies the incident and its routing: -You can also add or remove responders from this screen. +- **Responders** — who's assigned. Add or remove them from here. +- **Service**, **alert source**, and **escalation policy** — each links to its own page. +- **Status** — Triggered, Acknowledged, or Resolved, derived from the incident's state. +- **Severity** — critical, high, medium, or low. Edit it inline. -## Incident Metrics +Use **Acknowledge** or **Resolve** in the top-right to update the status; you can also unacknowledge or unresolve. -The detail page shows key incident fields such as: +## Incident metrics -- **Status** -- **Severity** -- **Suppressed Count** -- **Occurrence Count** -- **Triggered At** -- **Acknowledged At** -- **Resolved At** -- **Time to Resolve** +Alongside the header, the page shows the numbers that matter during response: -## Incident Summary +- **Occurrence count** and **suppressed count** +- **First seen**, **Triggered at**, **Acknowledged at**, **Resolved at** +- **Time to resolve** -The summary section contains the incident description and a link to the related alarm for deeper investigation. +## Manage the incident -## Activity Tabs +- **Responders** — add or remove the users assigned to the incident. +- **Severity** — change the severity; the change is recorded on the timeline. +- **Summary** — edit the incident description with the context of what happened and its impact. -At the bottom of the page, you can review: +![Incident responders panel](./images/incident-responders.png) -- **Notes:** Team notes and investigation context -- **Logs:** A timeline of assignments, acknowledgements, and other actions -- **Alarms:** The alarms attached to the incident +## Publish to a status page -![image](./images/incident-details-2.png) +When the affected service is linked to a status page, admins see a **Publish to status page** action on an open incident. It opens a dialog that pre-fills the page and affected components from the incident's service, so you can post a customer-facing incident in a couple of clicks. See [Post incidents and updates](../../status-pages/posting-incidents/). -Use **Acknowledge** or **Resolve** from the top-right corner to update the incident status. +![Publish to status page dialog](./images/incident-publish-status-page.png) + +## Linked SLOs + +When the affected service has at least one incident-based [SLO](../../../reliability/), a **Linked SLOs** panel appears below the standard service / alert source / escalation-policy labels. Each linked SLO shows as a card with its name, kind + target + window, and current compliance %. Clicking a card opens the [SLO detail](../../../reliability/slo-detail/) page. + +The panel only renders when at least one incident-based SLO exists on the affected service — incidents on services without reliability tracking show no panel. + +## Activity timeline + +The activity timeline is the incident's audit trail. It merges two kinds of entry into one time-ordered stream: + +- **Lifecycle events** — Created, Acknowledged, Unacknowledged, Resolved, Unresolved, Assigned, Unassigned, Severity changed, and Summary updated. +- **Notification deliveries** — one entry per attempt to reach a recipient, tagged with its **channel** (email, SMS, voice, Slack, MS Teams, webhook, Jira, or SNS) and a **status**: queued, sent, delivered, failed, skipped — unverified (the recipient had no verified phone), or skipped — no credits. + +Delivery entries also note which **step** paged the recipient, the **retry attempt**, and the **fallback order** — so you can trace exactly how the escalation policy unfolded, attempt by attempt. Read it top to bottom and you get the whole story: when it triggered, who was paged on which channel, what was delivered, who acknowledged, and when it resolved. + +![Incident activity timeline](./images/incident-activity-timeline.png) + +## Alarms + +The **Alarms** tab lists the raw alerts that fed this incident, each with its received time and whether it was suppressed. Expand a row to see the full payload that arrived from the alert source — useful for confirming what the source actually sent. + +## Notes + +The **Notes** tab is for team commentary: investigation findings, links, and context. Add a note and it's timestamped against your name, building a running record next to the automated timeline. diff --git a/src/content/docs/docs/incident-management/incidents/index.mdx b/src/content/docs/docs/incident-management/incidents/index.mdx index c64f4c10..47adb765 100644 --- a/src/content/docs/docs/incident-management/incidents/index.mdx +++ b/src/content/docs/docs/incident-management/incidents/index.mdx @@ -1,6 +1,6 @@ --- title: "Incidents" -description: "Review incident status, team workload, and assigned incidents in KloudMate." +description: "Review incident status and team workload, and work the full incident list with filters and bulk actions." sidebar: label: "Overview" order: 1 @@ -15,6 +15,8 @@ Incidents in KloudMate move through three core statuses: - **Acknowledged:** A responder has taken ownership and is actively reviewing it. - **Resolved:** The incident has been fixed and closed. +Status isn't a field you set directly — it's derived from whether the incident has been acknowledged and resolved. Acknowledging a triggered incident moves it to Acknowledged; resolving it moves it to Resolved. You can reverse either action too (unacknowledge, unresolve). + At the top of the page, you can review: - Total incident count @@ -27,3 +29,23 @@ At the top of the page, you can review: The **My Incidents** list shows incidents currently assigned to you, including severity, responders, triggered time, and current status. Use **View All** to open the full incident list across the workspace. + +## The incident list + +The full list is where you work through incidents at scale. Each row shows the incident's **ID** — its stable key, which links to the detail page — along with the title, service, severity, triggered time, responders, and status. + +Filter the list by: + +- **Status** — triggered, acknowledged, or resolved. +- **Service**, **alert source**, and **escalation policy**. +- **Assigned to me** — only incidents you're a responder on. +- **Show Resolved** — include resolved incidents, which are hidden by default. + +You can also act in bulk: select several incidents and acknowledge, resolve, or change the severity of all of them at once — handy when one root cause spawned a wave of incidents. + +![Incident list with filters and bulk actions](./images/incidents-list-filters.png) + +## Related + +- [Incident Details](./incident-details/) — the activity timeline, responders, and status actions. +- [Services](../services/) · [Alert Sources](../integrations/) · [Escalation Policies](../escalation-policy/) — what the list filters on. diff --git a/src/content/docs/docs/incident-management/index.mdx b/src/content/docs/docs/incident-management/index.mdx index f8720193..2f3076c8 100644 --- a/src/content/docs/docs/incident-management/index.mdx +++ b/src/content/docs/docs/incident-management/index.mdx @@ -13,11 +13,19 @@ Use Incident Management to organize incident response workflows, route alerts to - [Incident Lifecycle](./incident-lifecycle/) - [Incidents](./incidents/) - [Services](./services/) +- [Alert Sources](./integrations/) - [Escalation Policies](./escalation-policy/) -- [Integrations](./integrations/) +- [On-Call Schedules](./oncall/) +- [Routing Rules](./routing-rules/) +- [Status Pages](./status-pages/) - [Slack Integration](./slack-integration/) +:::note +**Two features share the name "Routing Rules."** The [Routing Rules](./routing-rules/) listed here decide how an incoming alert becomes an incident — which service, escalation policy, and severity it gets, and whether to suppress it. To route alert *notifications* to channels like Slack, see [Alerts → Routing Rules](../alerts/routing-rules/) instead. +::: + ## Related Resources -- [Alarms & Notifications](../alarms/) +- [Alerts & Notifications](../alerts/) +- [Reliability & SLOs](../reliability/) - [KloudMate Assistant](../kloudmate-assistant/) diff --git a/src/content/docs/docs/incident-management/integrations/_meta.json b/src/content/docs/docs/incident-management/integrations/_meta.json index 86368803..6ee159b4 100644 --- a/src/content/docs/docs/incident-management/integrations/_meta.json +++ b/src/content/docs/docs/incident-management/integrations/_meta.json @@ -1,4 +1,4 @@ { - "label": "Integrations", - "order": 7 + "label": "Alert Sources", + "order": 6 } diff --git a/src/content/docs/docs/incident-management/integrations/adding-integrations.mdx b/src/content/docs/docs/incident-management/integrations/adding-integrations.mdx index 3a11b55d..ba006a28 100644 --- a/src/content/docs/docs/incident-management/integrations/adding-integrations.mdx +++ b/src/content/docs/docs/incident-management/integrations/adding-integrations.mdx @@ -1,60 +1,132 @@ --- -title: "Add an Integration" -description: "Create KloudMate Alarm, CloudWatch Alarm, or webhook integrations for Incident Management." +title: "Add an Alert Source" +description: "Create KloudMate Alert group, CloudWatch Alarm, or webhook alert sources, map webhook payloads to incidents with Liquid templates, and POST alerts to the inbound URL." sidebar: - label: "Add an Integration" + label: "Add an Alert Source" order: 2 --- -Use the steps below to create a new incident integration. +import { Steps } from '@astrojs/starlight/components'; -1. Click **Add Integration** from the Integrations page. +An alert source receives alerts and turns them into incidents on a service. This page covers creating each type, mapping a webhook's payload to incident fields with templates, and sending alerts to its inbound URL. -![image](./images/adding-integrations-1.jpeg) +## Create an alert source -2. Select an **Integration Type**. +<Steps> +1. Click **Add** from the Alert Sources page. -Available types: +2. Choose a **type**: KloudMate Alert group, CloudWatch Alarm, or Webhook. -- **KloudMate Alarm** -- **CloudWatch Alarm** -- **Webhook** +3. Pick the **service** the source opens incidents on. This is required. -![image](./images/adding-integrations-1.png) +4. Enter a **name** and **description**. -:::note - KloudMate Alarm and CloudWatch Alarm integrations use predefined templates. Webhook integrations let you configure title, grouping, and auto-resolution logic manually. -::: +5. *(Optional)* Choose an **escalation policy** override. Leave it unset to use the service's default. -## Create a KloudMate Alarm or CloudWatch Alarm Integration +6. For a **Webhook**, configure the templates (below). KloudMate Alert group and CloudWatch Alarm sources use built-in templates, so there's nothing more to configure. -1. Select the service. -2. Enter the integration name and description. -3. Optionally choose an escalation policy override. -4. Click **Save**. +7. Click **Save**. +</Steps> -![image](./images/adding-integrations-2.png) +![Alert source type selector](./images/adding-integrations-1.png) -:::note - If no escalation policy is selected, the service default is applied automatically. -::: +After saving, you land on the source's details page, where you can copy its inbound URL. -## Create a Webhook Integration +## The inbound URL -1. Select the service. -2. Enter the integration name and description. -3. Configure the **Title Template**. -4. Configure the **Grouping Template**. -5. Configure the **Auto Resolve Template**. -6. Select an escalation policy if needed. -7. Click **Save**. +Every alert source gets its own hashed endpoint: + +``` +https://<your-api-host>/im/hooks/<hash> +``` + +Copy it from the source's details page with **Copy source URL**, then point your sender at it. The hash is what ties an incoming payload to this source. + +**CloudWatch** sources receive alarms through **Amazon SNS**: point an SNS topic at the URL, and KloudMate confirms the subscription automatically the first time it's hit. CloudWatch wraps the alarm JSON inside the SNS message, and KloudMate unwraps it for you. + +## Send an alert + +For a **Webhook** source, send your alert as a JSON `POST` to the inbound URL. The body can be any shape — your [templates](#map-a-webhook-payload-with-templates) decide how each field maps to the incident. + +```bash +curl -X POST https://<your-api-host>/im/hooks/<hash> \ + -H "Content-Type: application/json" \ + -d '{ + "alert": "High error rate on checkout-api", + "state": "firing", + "priority": "high", + "fingerprint": "checkout-api-error-rate", + "summary": "Error rate is 12% over the last 5 minutes", + "runbook": "https://runbooks.example.com/checkout-errors" + }' +``` + +KloudMate responds with the incident it opened or updated. Sending another alert that maps to the same [grouping key](#how-grouping-and-auto-resolve-behave) folds onto that incident instead of opening a duplicate; send one your [auto-resolve template](#map-a-webhook-payload-with-templates) reads as resolved to close it. + +## Map a webhook payload with templates + +A webhook can come from anywhere, so you tell KloudMate how to read it. Five fields take [Liquid](https://liquidjs.com/) templates that render against the incoming JSON: + +| Field | Sets | Notes | +|---|---|---| +| **Title template** | The incident title | | +| **Incident description template** | The incident description | Markdown — `**bold**`, `_italic_`, `` `code` ``, bullets, and `[links](url)` render in the incident. | +| **Severity template** | The severity | Must resolve to `critical`, `high`, `medium`, or `low`. | +| **Grouping template** | The grouping key | Alerts that render the same key fold into one incident. Leave it empty to group by title. | +| **Auto resolve template** | Whether to auto-resolve | Resolve it to `true`, `ok`, or `1` to close the matching incident automatically. | + +### A worked example + +For the payload from [Send an alert](#send-an-alert), these templates open an incident titled *High error rate on checkout-api* at **high** severity: + +| Field | Template | Renders to | +|---|---|---| +| Title template | `{{ alert }}` | High error rate on checkout-api | +| Incident description template | `{{ summary }} — [Runbook]({{ runbook }})` | Error rate is 12% over the last 5 minutes — Runbook | +| Severity template | `{{ priority }}` | high | +| Grouping template | `{{ fingerprint }}` | checkout-api-error-rate | +| Auto resolve template | `{% if state == "resolved" %} true {% else %} false {% endif %}` | false | + +Send the same payload with `"state": "resolved"` and the auto-resolve template renders `true`, closing the matching incident. + +### Liquid syntax + +The template tester's **Syntax reference** covers the constructs you'll use: + +- **Variables** — `{{ alarmName }}` renders a value from the payload. +- **Conditions** — `{% if status == "Normal" %}Resolved{% else %}Triggered{% endif %}`, using `==` and `!=`. +- **Loops** — `{% for item in items %}{{ item.evalString }}{% endfor %}` iterates an array. +- **Trusted HTML** — `{{ html | raw }}` outputs unescaped HTML, only when you trust the source. + +An unknown variable renders as empty rather than erroring, so a template won't break if a field is missing from the payload. + +### Test before you save + +The **template tester** checks a template against a real payload without waiting for an alert: + +<Steps> +1. Pick a **sample payload** — a CloudWatch alarm or a KloudMate alarm — or paste your own JSON. + +2. Edit the **template**. + +3. Click **Test** to see the rendered output. For the auto-resolve template, the tester also tells you whether the incident would auto-resolve. +</Steps> + +![Webhook alert source template fields](./images/webhook-template-fields.png) + +![Template tester with rendered output](./images/webhook-template-tester.png) + +![Liquid syntax reference](./images/webhook-liquid-reference.png) -![image](./images/adding-integrations-3.png) +### How grouping and auto-resolve behave -After the integration is created, you will be taken to the integration details screen where you can copy the integration URL or edit/archive the integration. +**Grouping** keeps repeat alerts together. While an incident is open, every alert that renders the same grouping key folds onto it instead of opening a duplicate. Leave the grouping template empty and KloudMate groups by the rendered title instead. -![image](./images/adding-integrations-2.jpeg) +**Auto resolve** closes the matching open incident when the auto-resolve template renders `true`, `ok`, or `1`. A resolving alert that matches no open incident is ignored — KloudMate never opens an incident from a recovery. Once an incident is resolved, the next firing alert with the same grouping key opens a *new* incident, linked as a recurrence of the one before it. -## Related Resources +## Related -- [Integrating with KloudMate Alarms](../integrating-with-kloudmate-alarms/) +- [Alert Sources overview](../) — types and details. +- [Integrate with KloudMate Alerts](../integrating-with-kloudmate-alerts/) — wire KloudMate alerts into IM through a channel and routing rule. +- [Integrate with AWS CloudWatch Alarms](../integrating-with-aws-cloudwatch/) — route AWS CloudWatch alarms to KloudMate via SNS. +- [Routing Rules](../../routing-rules/) — decide which service, escalation policy, and severity an incident gets. diff --git a/src/content/docs/docs/incident-management/integrations/images/adding-integrations-1.png b/src/content/docs/docs/incident-management/integrations/images/adding-integrations-1.png index 5a3cb2cd..19149f70 100644 Binary files a/src/content/docs/docs/incident-management/integrations/images/adding-integrations-1.png and b/src/content/docs/docs/incident-management/integrations/images/adding-integrations-1.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/alert-sources-list.png b/src/content/docs/docs/incident-management/integrations/images/alert-sources-list.png new file mode 100644 index 00000000..cb01ddd4 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/alert-sources-list.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-1.png b/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-1.png new file mode 100644 index 00000000..09564387 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-1.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-2.png b/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-2.png new file mode 100644 index 00000000..626e5c90 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/cloudwatch-alert-source-2.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/kloudmate-incidents-channel-form.png b/src/content/docs/docs/incident-management/integrations/images/kloudmate-incidents-channel-form.png new file mode 100644 index 00000000..83fa91b5 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/kloudmate-incidents-channel-form.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/routing-rule-km-incidents.png b/src/content/docs/docs/incident-management/integrations/images/routing-rule-km-incidents.png new file mode 100644 index 00000000..4e0268d7 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/routing-rule-km-incidents.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/webhook-liquid-reference.png b/src/content/docs/docs/incident-management/integrations/images/webhook-liquid-reference.png new file mode 100644 index 00000000..2a9cfc92 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/webhook-liquid-reference.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/webhook-template-fields.png b/src/content/docs/docs/incident-management/integrations/images/webhook-template-fields.png new file mode 100644 index 00000000..56856f38 Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/webhook-template-fields.png differ diff --git a/src/content/docs/docs/incident-management/integrations/images/webhook-template-tester.png b/src/content/docs/docs/incident-management/integrations/images/webhook-template-tester.png new file mode 100644 index 00000000..286a1bfe Binary files /dev/null and b/src/content/docs/docs/incident-management/integrations/images/webhook-template-tester.png differ diff --git a/src/content/docs/docs/incident-management/integrations/index.mdx b/src/content/docs/docs/incident-management/integrations/index.mdx index a3a39948..c29c10fa 100644 --- a/src/content/docs/docs/incident-management/integrations/index.mdx +++ b/src/content/docs/docs/incident-management/integrations/index.mdx @@ -1,32 +1,44 @@ --- -title: "Integrations" -description: "Create and manage incident integrations that route alerts into KloudMate." +title: "Alert Sources" +description: "Alert sources receive alerts from KloudMate, CloudWatch, and webhooks, and turn them into incidents." sidebar: label: "Overview" order: 1 --- -Integrations are endpoints that accept alerts from external systems and turn them into incidents inside KloudMate. +Alert sources bring alerts into Incident Management. Each one accepts alerts from an external system, maps the incoming payload to incident fields, and opens incidents on a service. -Each integration belongs to a service and can use a dedicated escalation policy or inherit the service default. +:::note +Alert sources were previously called **Integrations**. The feature is the same; only the name changed. +::: -## Integrations Overview +Every alert source belongs to a **service** (required) and can set a default **escalation policy** (optional). Leave the policy unset and the source inherits the service's default. -Open **Incident Management -> Integrations** to view the list of configured integrations. +## Alert source types -![image](./images/integrations-1.jpeg) +| Type | Use it for | +|---|---| +| **KloudMate Alert group** | Opening incidents from KloudMate alerts. | +| **CloudWatch Alarm** | Receiving AWS CloudWatch alarms, delivered through SNS. | +| **Webhook** | Any system that can POST JSON, with your own templates mapping the payload to incident fields. | -Each row shows the integration name, description, type, related service, and assigned escalation policy. +KloudMate Alert group and CloudWatch Alarm sources use built-in templates. To integrate them, see [Integrate with KloudMate Alerts](./integrating-with-kloudmate-alerts/) and [Integrate with AWS CloudWatch Alarms](./integrating-with-aws-cloudwatch/). For custom webhooks, see [Add an alert source](./adding-integrations/). -:::note - Each integration can use only one escalation policy at a time. -::: +## The alert sources list + +Open **Incident Management → Alert Sources** to see every source with its name, description, type, service, and escalation policy. -## Integration Details +![Alert sources list](./images/alert-sources-list.png) -Open any integration to: +## Alert source details + +Open a source to: - Enable or disable it -- Copy the integration URL +- Copy its inbound **source URL** — a unique, hashed endpoint - Edit its configuration - Archive it + +:::note +Each alert source uses one escalation policy at a time — its own override, or the service default. +::: diff --git a/src/content/docs/docs/incident-management/integrations/integrating-with-aws-cloudwatch.mdx b/src/content/docs/docs/incident-management/integrations/integrating-with-aws-cloudwatch.mdx new file mode 100644 index 00000000..62ba3af3 --- /dev/null +++ b/src/content/docs/docs/incident-management/integrations/integrating-with-aws-cloudwatch.mdx @@ -0,0 +1,125 @@ +--- +title: "Integrate with AWS CloudWatch Alarms" +description: "Route AWS CloudWatch alarm breaches to KloudMate to trigger escalation policies, deduplicate alerts, and automatically resolve incidents." +sidebar: + label: "AWS CloudWatch" + order: 4 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Connect your AWS CloudWatch alarms to KloudMate Incident Management to route alarm transitions to your escalation policies and notify your team via Slack, email, SMS, or voice. KloudMate groups incoming alarm events, dedupes them to prevent alert noise, and automatically resolves incidents when the alarms return to normal. + +This integration uses an **AWS Simple Notification Service (SNS)** topic to route CloudWatch state changes to a KloudMate webhook endpoint. + +## Prerequisites + +Before you start, make sure you have: +- An [Incident Management service](../../services/) configured in KloudMate. +- Permissions in AWS to create SNS topics, subscriptions, and CloudWatch alarms. + +## Step 1: Create a CloudWatch alert source in KloudMate + +Create a unique ingestion endpoint in KloudMate to receive your CloudWatch alerts. + +<Steps> +1. Log into the KloudMate Console and open **Incident Management → Alert Sources**. + +2. Click **Add** (or navigate to a specific [Service](../../services/) and click **Create Alert Source** under details). + +3. Set the **Type** to `CloudWatch Alarm`. + +4. Select the **Service** you want to associate with these alarms. + +5. Enter a **Name** and **Description** for the alert source. + +6. Click **Save**. + +7. From the details page of your new alert source, copy the **Source URL** (e.g. `https://api.kloudmate.com/im/hooks/cw-a1b2c3d4e5f6`). +</Steps> + +![Create CloudWatch alert source](./images/cloudwatch-alert-source-1.png) + +![CloudWatch alert source URL](./images/cloudwatch-alert-source-2.png) + +## Step 2: Set up an AWS SNS topic + +CloudWatch alarms publish their status changes to an Amazon Simple Notification Service (SNS) topic, which then pushes them to KloudMate. + +<Steps> +1. Log into the **AWS Management Console** and open the **Amazon SNS** console. + +2. In the left navigation, click **Topics**, then click **Create topic**. + +3. Choose **Standard** as the topic type. + +4. Enter a **Name** (for example, `kloudmate-incidents-trigger`) and optional **Display name**. + +5. Keep all other default settings and click **Create topic**. +</Steps> + +## Step 3: Create an HTTPS subscription + +Subscribe your KloudMate source URL to your AWS SNS topic. + +<Steps> +1. In your newly created SNS topic page, click **Create subscription**. + +2. Set the **Protocol** to `HTTPS`. + +3. In the **Endpoint** field, paste the **Source URL** you copied from KloudMate in Step 1. + +4. Keep the default settings and click **Create subscription**. +</Steps> + +:::tip[Automatic Confirmation] +AWS SNS sends a subscription confirmation request to the endpoint immediately. KloudMate automatically detects and approves this handshake. Refresh your AWS console after a few seconds — the subscription status changes from **PendingConfirmation** to **Confirmed** without any manual action. +::: + +## Step 4: Configure your CloudWatch alarms + +Configure your AWS CloudWatch alarms to notify the SNS topic when they transition to an alarm state and when they recover. + +<Steps> +1. Open the **Amazon CloudWatch** console and navigate to **Alarms → All alarms**. + +2. Select an existing alarm and click **Edit**, or click **Create alarm**. + +3. In the **Configure actions** step: + - Under **Alarm state trigger**, select **In alarm**. + - Under **Send a notification to the following SNS topic**, select **Select an existing SNS topic** and choose the topic you created in Step 2. + +4. Click **Add notification** to configure a recovery action: + - Under **Alarm state trigger**, select **OK**. + - Under **Send a notification to the following SNS topic**, select the same SNS topic. + +5. Click **Update alarm** (or complete the alarm creation steps) to save your changes. +</Steps> + +## How it works + +KloudMate processes incoming CloudWatch alarm payloads automatically to keep your incidents organized and up to date. + +### Grouping and deduplication + +To prevent alert fatigue and duplicate notifications, KloudMate groups incoming alert events: + +- When an alarm enters the `ALARM` state, AWS SNS pushes the event payload to KloudMate. +- KloudMate generates a unique **Group ID** by hashing the `AlarmName` from the payload. +- **If no open incident exists** for this Group ID, KloudMate opens a new incident, processes your [Routing Rules](../../routing-rules/), and triggers the configured escalation policy. +- **If an open incident already exists** for this Group ID, KloudMate logs the new alarm event under the existing incident and increments its suppression count. No duplicate incidents or notifications are created. + +### Automatic resolution + +When the underlying issue clears, KloudMate resolves the incident automatically: + +- When the CloudWatch alarm transitions back to `OK`, AWS SNS pushes the state change to KloudMate. +- KloudMate inspects the payload (`NewStateValue == "OK"`), identifies it as a recovery signal, and automatically marks the active incident as **Resolved**. +- This immediately halts the escalation policy, stopping any further notifications. + +## Related + +- [Alert Sources overview](../) — types and details of all supported alert sources. +- [Add an Alert Source](../adding-integrations/) — how to set up sources and map custom webhook payloads. +- [Routing Rules](../../routing-rules/) — decide which service, escalation policy, and severity an incident gets. +- [Escalation Policies](../../escalation-policy/) — define who to page and when. diff --git a/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alarms.mdx b/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alarms.mdx deleted file mode 100644 index efb72825..00000000 --- a/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alarms.mdx +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: "Integrate with KloudMate Alarms" -description: "Send KloudMate alarm notifications into Incident Management through a webhook integration." -sidebar: - label: "Integrate with KloudMate Alarms" - order: 3 ---- - -To send alarm-triggered notifications into Incident Management, connect a webhook integration to a notification channel and notification policy. - -## Step 1: Copy the Integration URL - -1. Open **Incidents -> Integrations**. -2. Select an integration of type **KloudMate Alarm**. -3. Copy the integration URL from the integration details page. - -![image](./images/integrating-with-kloudmate-alarms-1.jpeg) - -If you have not created the integration yet, start with [Add an Integration](../adding-integrations/). - -## Step 2: Create a Notification Channel - -1. Open **Settings -> Notification Channels**. -2. Click **Create**. -3. Select **Webhook**. -4. Enter a name and paste the integration URL. -5. Submit the channel. - -![image](./images/integrating-with-kloudmate-alarms-1.png) -![image](./images/integrating-with-kloudmate-alarms-2.jpeg) -![image](./images/integrating-with-kloudmate-alarms-2.png) - -## Step 3: Create a Notification Policy - -1. Open **Settings -> Notification Policies**. -2. Click **Create**. -3. Enter a policy name. -4. Select the webhook notification channel. -5. Add conditions for **Alarm** or **Issue**. -6. Save the policy. - -![image](./images/integrating-with-kloudmate-alarms-3.jpeg) -![image](./images/integrating-with-kloudmate-alarms-3.png) -![image](./images/integrating-with-kloudmate-alarms-4.jpeg) -![image](./images/integrating-with-kloudmate-alarms-4.png) - -:::note - Alerts matching this notification policy will be sent into the selected Incident Management integration and follow its escalation policy. -::: diff --git a/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alerts.mdx b/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alerts.mdx new file mode 100644 index 00000000..a2017d70 --- /dev/null +++ b/src/content/docs/docs/incident-management/integrations/integrating-with-kloudmate-alerts.mdx @@ -0,0 +1,86 @@ +--- +title: "Integrate with KloudMate Alerts" +description: "Open Incident Management incidents from KloudMate alerts using a KloudMate Incidents notification channel and a routing rule." +sidebar: + label: "KloudMate Alerts" + order: 3 +--- + +import { Steps } from '@astrojs/starlight/components'; + +To open an Incident Management incident whenever an alert fires, you wire two things together: a **KloudMate Incidents** notification channel (which bridges KloudMate alerts into the IM service) and a **Routing Rule** (which decides which alerts feed the channel). + +This replaces the older "webhook channel + notification policy with Alarm condition" flow. The KloudMate Incidents channel handles the bridge plumbing for you; routing rules give you label-level control over which alerts trigger which IM services. + +:::note +The **Routing Rule** on this page is an [Alerts routing rule](../../../alerts/routing-rules/) — it routes alert *notifications* to channels. Don't confuse it with [Incident Management → Routing Rules](../../routing-rules/), which decides how an alert becomes an incident once it has reached IM. +::: + +## Prerequisites + +- An [IM service](../../services/) you want incidents to open against. +- A workspace admin role (you need it to create channels and routing rules). + +## Step 1 — Create a KloudMate Incidents channel + +<Steps> +1. Open **Settings → Notification Channels → Create**. + +2. Pick **KloudMate Incidents** as the channel type. + +3. Pick the IM service you want to link from the **Linked IM service** dropdown. + +4. Click **Submit**. +</Steps> + +KloudMate creates the IM integration row under your service automatically and points the new channel at the integration's signed loop URL. You don't need to copy URLs or paste them anywhere. See [Notification Channels](../../../platform/settings/notification-channels/#kloudmate-incidents) for the channel-side details. + +![Create KloudMate Incidents channel](./images/kloudmate-incidents-channel-form.png) + +## Step 2 — Route alerts to the channel + +<Steps> +1. Open **Alerts → Routing Rules → Create**. + +2. Name the rule and set a priority (lower wins). + +3. Add **matchers** that identify which alerts should feed this IM service, e.g. `service equals payments` to route every payments alert to the payments IM service. + +4. Select **group-by** keys (e.g. `service`, `env`) so related alerts fold into a single notifiable group. + +5. Select the **KloudMate Incidents channel** you created in Step 1 as a destination. + +6. Optionally enable **Auto-RCA** if you want KloudMate's AI investigator to attach a summary to the incident automatically. + +7. Click **Save**. +</Steps> + +See [Routing Rules](../../../alerts/routing-rules/) for the full rule editor walk-through. + +![Routing rule with KloudMate Incidents destination](./images/routing-rule-km-incidents.png) + +## Step 3 — Verify + +Trigger a matching alert. Within the routing rule's `group_wait` (default 30 seconds), an incident opens on the linked IM service under the configured escalation policy. + +Things to check: + +- The incident shows up on the IM service's incidents tab. +- The originating [Alert Group](../../../alerts/alert-groups/) on the **Alerts → Groups** page lists the KloudMate Incidents channel under its Notifications tab with the new incident's ID. +- If Auto-RCA was enabled, the investigation summary attaches to both the group and the incident. + +## Migrating from the older flow + +If you used to wire this up via **Webhook channel + Notification Policy with Alarm condition**, the upgrade migrated each policy to a routing rule. No recreation is required — but the recommended path going forward is the KloudMate Incidents channel, not a raw webhook, because: + +- KloudMate Incidents channels carry HMAC signing automatically. +- The KloudMate Incidents receiver knows how to dedupe against existing incidents and append RCA summaries when they arrive late. +- Routing-rule label matchers give you finer control than tag-based policies did. + +## Related + +- [Alerts Routing Rules](../../../alerts/routing-rules/) — the rule editor used here (routes notifications). +- [IM Routing Rules](../../routing-rules/) — the different feature that turns alerts into incidents. +- [Alert Groups](../../../alerts/alert-groups/) — what gets routed. +- [Notification Channels](../../../platform/settings/notification-channels/#kloudmate-incidents) — KloudMate Incidents channel setup. +- [IM Services](../../services/) — the destination side. diff --git a/src/content/docs/docs/incident-management/oncall/_meta.json b/src/content/docs/docs/incident-management/oncall/_meta.json new file mode 100644 index 00000000..417b5026 --- /dev/null +++ b/src/content/docs/docs/incident-management/oncall/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "On-Call Schedules", + "order": 8 +} diff --git a/src/content/docs/docs/incident-management/oncall/creating-a-schedule.mdx b/src/content/docs/docs/incident-management/oncall/creating-a-schedule.mdx new file mode 100644 index 00000000..70f42f40 --- /dev/null +++ b/src/content/docs/docs/incident-management/oncall/creating-a-schedule.mdx @@ -0,0 +1,125 @@ +--- +title: "Create an On-Call Schedule" +description: "Set up an on-call schedule: start from a template, then shape rotations and coverage in the calendar editor." +sidebar: + label: "Create a Schedule" + order: 2 +--- + +import { Steps } from '@astrojs/starlight/components'; + +The fastest way to understand on-call schedules is to build the common one: a single rotation that hands the on-call duty between a few people every week. This page walks that end to end through the template picker and the calendar editor, then covers each rotation field so you can adjust it. + +:::note[Availability] +On-Call Schedules are a paid Incident Management feature. If you don't see **On-Call Schedules** in the Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## Create a schedule + +<Steps> +1. Open **Incident Management → On-Call Schedules** and click **New schedule**. + +2. Pick a starting point from the **Quick start** template picker (see below). For a single weekly rotation, choose **Weekly handoff**. + +3. Answer the short **guided wizard** — the timezone, who covers, and when to hand off. It pre-creates the rotation for you and drops you into the calendar editor. + +4. Review the **calendar**. It lays out every rotation's shifts, color-coded per person, so you can confirm the handoffs land where you expect. + +5. Adjust the **schedule name** (the pencil in the top bar) and the **timezone** if needed, then open any rotation to fine-tune it. + +6. Click **Save schedule**. +</Steps> + +![New on-call schedule template picker](./images/schedule-quick-start.png) + +## Quick start: pick a template + +**New schedule** opens a **Quick start** picker with four cards. Each one is just a head start — everything it creates is fully editable afterward in the calendar editor. + +- **Weekly handoff** — one rotation that rotates through its members, handing off once a week. The 80% case. +- **Workweek / weekend** — two complementary rotations: a weekday team and a separate nights-and-weekends rotation. +- **Follow the sun** — one rotation per region, each in its own timezone, so coverage hands around the globe. You add as many **regions** as you need; each region has its own name, timezone, members, and active days/times, and each becomes a rotation. +- **Custom** — an empty editor. No rotations are pre-created; you add them yourself. + +Picking any template except **Custom** opens a short **guided wizard** that asks plain-language questions — the timezone, who's covering, and when to hand off — and pre-creates the matching rotation(s). **Custom** skips the wizard and drops you straight into an empty calendar editor. + +## The schedule editor + +The editor is **calendar-first**. The main surface is a week/day **calendar** showing every rotation's shifts, color-coded per person, so the whole schedule reads at a glance. + +- A right-hand **Rotations** panel lists each rotation with a one-line summary and its timezone, plus a **+ Create rotation** button. +- An **Overrides** panel sits below it for one-off swaps. See [Overrides](../overrides/). +- The top bar carries the **schedule name** (click the pencil to rename), a **timezone** selector, and **Save schedule**. + +Each calendar event shows the responder's name with the rotation name beneath it. Click an event to open a details popover — the person, the rotation, the shift's start and end, and an **Edit rotation** shortcut. + +![On-call schedule calendar editor](./images/schedule-calendar-editor.png) + +## Create a rotation + +Open a rotation from the **Rotations** panel — **+ Create rotation** for a new one, or an existing row to edit it — to open the **rotation drawer**. + +<Steps> +1. Give the rotation a **name**. It labels the rotation's shifts on the calendar and its row in the Rotations panel. + +2. Set the **priority** if this rotation should win an overlap with another. Leave it at **0** — the default — to keep the rotation additive with the rest; a higher number takes over lower ones where they overlap. See [Rotation priority](../rotations-and-coverage/#rotation-priority). + +3. Set the **timezone** — leave it on **Use schedule timezone**, or override it for this rotation (the basis for follow-the-sun). All of this rotation's times are evaluated in its effective timezone. + +4. Set the **coverage**. Leave it at **24/7** for always-on, or choose **Custom** and set **Active days** (pick days, or a day range) and **Active times** (the **All day** toggle, or a from–to window). See [Rotations and coverage](../rotations-and-coverage/). + +5. Choose the **shift assignment**: **Handoff and rotate to next member** (one person on call per shift) or **All members are on-call for each shift** (everyone in the rotation at once). + +6. For a rotating shift, set the **handoff** — every *N* hours, days, or weeks. For a weekly cadence, pick the handoff **weekday** and **time**; a **start date** anchors the first shift. + +7. Add **members** in the order they should take a shift. Order matters: the first person covers the first shift, the second the next, and so on. Reordering them later changes who's on call when. + +8. Save the rotation. It appears on the calendar immediately so you can check it before saving the schedule. +</Steps> + +![Rotation drawer with rotation settings](./images/schedule-rotation-drawer.png) + +## Example: a weekly handoff across three people + +This is the 80% case — one rotation that rotates weekly, handing off every Monday at 09:00, across three members: + +| Field | Value | +|---|---| +| Shift assignment | Handoff and rotate to next member | +| Handoff | Weekly, Mondays at 09:00 | +| Coverage | 24/7 | +| Members (in order) | Alice, Bob, Carol | + +Alice covers this week. Bob takes over next Monday at 09:00, Carol the Monday after, then it loops back to Alice. The handoff always happens at 09:00 in the rotation's timezone. + +Because the handoff **weekday** is its own field, you set it directly — pick **Monday** and the rotation hands off on Mondays, no matter which date you started it on. + +## Rotation field reference + +Every rotation has the same fields. The defaults below are what a fresh rotation starts with. + +| Field | Values | Default | Notes | +|---|---|---|---| +| Name | Text | — | Labels the rotation's shifts on the calendar and its row in the Rotations panel. | +| Priority | Whole number, 0 or higher | 0 | Where rotations overlap, a higher priority takes over and hides lower ones; equal priority stays additive. | +| Timezone | Use schedule timezone, or any timezone | Use schedule timezone | The basis for this rotation's handoff and coverage times. Override it for follow-the-sun. | +| Coverage | 24/7, Custom | 24/7 | When the rotation is active. Custom adds Active days and Active times. | +| Active days | Selected weekdays, or a day range | — | Custom coverage only. The days a shift can start. | +| Active times | All day, or a from–to window | All day | Custom coverage only. The hours within the active days. | +| Shift assignment | Handoff and rotate to next member, All members are on-call for each shift | Handoff and rotate | One person per shift (advancing each handoff), or everyone in the rotation at once. | +| Handoff | Every *N* hours / days / weeks (weekly: on a weekday at a time) | Weekly | How often the on-call member changes. Applies only when the rotation hands off. | +| Start date | Date | Today | Anchors the first shift, together with the handoff time. | +| Members | Ordered list of users | — | Shift order follows this list; the first member takes the first shift. | + +## Read the calendar + +The calendar is the quickest check that a schedule does what you meant. It lays out who's on call across the coming weeks — every rotation's shifts, color-coded per person, with the exact handoff moments. Where two rotations overlap, you'll see both responders on the calendar at once — the additive default. If you gave one rotation a higher **priority**, the lower one drops out of the overlap instead, so a shift can disappear from the calendar on purpose. + +If a handoff lands on the wrong day or the wrong person leads, open the rotation, fix the handoff weekday or the member order, and watch the calendar update before you save. The **Schedules** tab's coverage timeline is also where you confirm there are no gaps with nobody on call. + +![On-call schedule calendar preview](./images/schedule-calendar-preview.png) + +## Next + +- [Rotations and coverage](../rotations-and-coverage/) — add a second rotation or limit one to business hours. +- [Overrides](../overrides/) — hand off temporarily for a vacation or a swap. diff --git a/src/content/docs/docs/incident-management/oncall/images/oncall-schedules-tab.png b/src/content/docs/docs/incident-management/oncall/images/oncall-schedules-tab.png new file mode 100644 index 00000000..1bdcf43d Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/oncall-schedules-tab.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/override-dialog.png b/src/content/docs/docs/incident-management/oncall/images/override-dialog.png new file mode 100644 index 00000000..fb35ef09 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/override-dialog.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/rotation-coverage-settings.png b/src/content/docs/docs/incident-management/oncall/images/rotation-coverage-settings.png new file mode 100644 index 00000000..518f5699 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/rotation-coverage-settings.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-editor.png b/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-editor.png new file mode 100644 index 00000000..b2dc88c3 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-editor.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-preview.png b/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-preview.png new file mode 100644 index 00000000..0d623472 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-calendar-preview.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-overrides-list.png b/src/content/docs/docs/incident-management/oncall/images/schedule-overrides-list.png new file mode 100644 index 00000000..679080d2 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-overrides-list.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-quick-start.png b/src/content/docs/docs/incident-management/oncall/images/schedule-quick-start.png new file mode 100644 index 00000000..8c80d0c5 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-quick-start.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-rotation-drawer.png b/src/content/docs/docs/incident-management/oncall/images/schedule-rotation-drawer.png new file mode 100644 index 00000000..0445ed6f Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-rotation-drawer.png differ diff --git a/src/content/docs/docs/incident-management/oncall/images/schedule-split-rotations.png b/src/content/docs/docs/incident-management/oncall/images/schedule-split-rotations.png new file mode 100644 index 00000000..b7f2df78 Binary files /dev/null and b/src/content/docs/docs/incident-management/oncall/images/schedule-split-rotations.png differ diff --git a/src/content/docs/docs/incident-management/oncall/index.mdx b/src/content/docs/docs/incident-management/oncall/index.mdx new file mode 100644 index 00000000..336afb93 --- /dev/null +++ b/src/content/docs/docs/incident-management/oncall/index.mdx @@ -0,0 +1,71 @@ +--- +title: "On-Call Schedules" +description: "Build rotations that decide who's on call at any moment, using coverage windows, priority, per-rotation timezones, and overrides." +sidebar: + label: "Overview" + order: 1 +--- + +On-call schedules decide who's on call when an incident needs a person. Rather than wiring a specific name into an escalation policy, you point the policy — or a [routing rule](../routing-rules/) — at a schedule, and KloudMate resolves it to whoever is on call at that exact moment. + +:::note[Availability] +On-Call Schedules are a paid Incident Management feature. If **On-Call Schedules** isn't in your Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## How a schedule decides who's on call + +A schedule is a named wrapper around one or more **rotations**. Once the mental model clicks, every other page here falls into place. + +- **Schedule** — a named container with a default **timezone**. It holds one or more rotations. The timezone is the default basis for every time inside the schedule, though any rotation can override it. +- **Rotation** — the unit that actually puts a person on call. Each rotation answers four questions: + - **Members** — an ordered list of people. The first member takes the first shift. + - **Shift assignment** — either **Handoff and rotate to next member** (one person is on call per shift, and it advances to the next member at each handoff) or **All members are on-call for each shift** (everyone in the rotation is on call together). + - **Shift length / handoff** — how long each shift lasts before the on-call person changes: sub-day (for example 8h or 12h), daily, weekly, or every *N* hours, days, or weeks. (This only applies when the rotation hands off.) + - **Coverage** — when the rotation is active: **24/7**, or a **Custom** window built from active days and active times. + - A rotation can also carry its own **timezone** that overrides the schedule's — the basis for follow-the-sun. +- **Override** — a timed swap that replaces the on-call set for a fixed window. See [Overrides](./overrides/). + +### How rotations combine + +By default, **rotations add up**. Who's on call at any instant is the **union of every active rotation's current assignee(s)**, deduplicated by person. A rotation is *active* at a given moment when that moment falls inside its coverage window — a 24/7 rotation is always active. If three rotations are active at once, all three of their current responders are on call together, and an escalation step that targets the schedule notifies every one of them. + +This is what makes multi-responder and follow-the-sun coverage natural: + +- **Multi-responder** — run two rotations over the same hours and both responders are on call at the same time (for example, a primary plus a shadowing secondary). +- **Follow-the-sun** — give each region its own rotation in its own timezone, and coverage hands around the globe as each region's working hours come round. + +When you *don't* want both people on call for an overlap, give one rotation a higher **priority**. The higher priority takes over and hides the lower one for as long as they overlap. Every rotation sits at priority `0` until you change it, so schedules stay additive unless you opt in — see [Rotations and coverage](./rotations-and-coverage/#rotation-priority). + +:::tip[Overlapping coverage adds up — unless you set a priority] +When two rotations' coverage windows overlap, both current responders are on call by default. To hand the overlap to one of them, raise that rotation's **priority**: a higher number wins and hides the lower one while they overlap. Equal priority (the default) keeps both on call. +::: + +## See who's on call + +The **On-call** section has two tabs: + +- **My on-call** — your personal, cross-schedule view: whether you're on call right now and your upcoming shifts across *every* schedule you belong to. This is the page to check before you log off for the day. +- **Schedules** — a compact list of every schedule, each showing who's on call now, a roughly two-week coverage timeline, and the timezone the times are shown in. Use it to scan coverage at a glance and spot windows where no one is rostered. + +![On-call section with My on-call and Schedules tabs](./images/oncall-schedules-tab.png) + +## Mind the gaps + +Because rotations are additive rather than a fallback stack, **custom coverage windows can leave moments with no rotation active — and so nobody on call**. If an incident fires during such a gap, an escalation step that targets the schedule has no one to reach. + +For any schedule an escalation policy depends on, keep at least one **24/7 rotation** (or make sure your rotations' windows tile the clock with no gap). The **Schedules** tab's coverage timeline is the quickest way to spot an uncovered window before you rely on the schedule. See [Rotations and coverage](./rotations-and-coverage/) for worked examples. + +## How schedules connect to incidents + +A schedule doesn't notify anyone on its own. It becomes useful when something targets it: + +- An **escalation policy** step can target a schedule instead of a named user. See [Escalation Policies](../escalation-policy/). +- A **routing rule** can notify one or more schedules when an alert matches. See [Routing Rules](../routing-rules/). + +In both cases the schedule is resolved *at notification time* — KloudMate looks up who's on call at that instant, which may be **several people**, and notifies each of them through their own notify methods (email, SMS, or voice). Edit a rotation or add an override a minute before an incident fires, and the new result is what gets notified. + +## In this section + +- [Create a schedule](./creating-a-schedule/) — the template picker, the calendar editor, and a simple weekly rotation. +- [Rotations and coverage](./rotations-and-coverage/) — coverage windows, per-rotation timezones, and additive patterns for business-hours, follow-the-sun, and overnight coverage. +- [Overrides](./overrides/) — cover a vacation or a one-off swap without touching the rotations. diff --git a/src/content/docs/docs/incident-management/oncall/overrides.mdx b/src/content/docs/docs/incident-management/oncall/overrides.mdx new file mode 100644 index 00000000..4887bb86 --- /dev/null +++ b/src/content/docs/docs/incident-management/oncall/overrides.mdx @@ -0,0 +1,57 @@ +--- +title: "Overrides" +description: "Temporarily replace who's on call for a fixed window — to cover a vacation or a one-off swap — without changing the rotation." +sidebar: + label: "Overrides" + order: 4 +--- + +import { Steps } from '@astrojs/starlight/components'; + +An override puts a specific person on call for a fixed window, then restores the normal rotations. Reach for one when the rotations are right in general but wrong for a few days — someone's on vacation, swapping a shift, or covering a deploy. The rotations underneath are untouched; the override sits on top of them for its window only. + +:::note[Availability] +On-Call Schedules are a paid Incident Management feature. If you don't see **On-Call Schedules** in the Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## Add an override + +<Steps> +1. Open the schedule and start a new **override**. + +2. Pick the **user** who'll cover the window. + +3. Set the **start** and **end** — both are a date and time, interpreted in the schedule's timezone. + +4. Add an optional **reason** ("Bob PTO", "covering deploy") so the rest of the team knows why. + +5. Save. The override takes effect immediately for its window. +</Steps> + +An override **replaces the active on-call set** for the time it covers — whoever the rotations would otherwise put on call is set aside until it ends, and only the override's person is on call. If two overrides overlap, the one created most recently wins. + +![Add override dialog](./images/override-dialog.png) + +## Example: cover a vacation + +Bob is on call next week but away Wednesday and Thursday. Cover the gap with one override: + +| Field | Value | +|---|---| +| User | Dana | +| Start | Wednesday 00:00 | +| End | Friday 00:00 | +| Reason | Bob PTO | + +Dana is on call from Wednesday until Friday at 00:00; Bob resumes automatically when the override ends. Nothing about the weekly rotation changes — the next handoff still happens on schedule. + +## Remove an override + +Plans change. Open the schedule, find the override in its list, and delete it. The rotations underneath resume for that window as if the override had never existed. + +![Schedule detail with overrides list](./images/schedule-overrides-list.png) + +## Related + +- [Create a schedule](../creating-a-schedule/) — the rotations an override sits on top of. +- [Rotations and coverage](../rotations-and-coverage/) — the permanent coverage structure an override sits on top of. diff --git a/src/content/docs/docs/incident-management/oncall/rotations-and-coverage.mdx b/src/content/docs/docs/incident-management/oncall/rotations-and-coverage.mdx new file mode 100644 index 00000000..0091fd7c --- /dev/null +++ b/src/content/docs/docs/incident-management/oncall/rotations-and-coverage.mdx @@ -0,0 +1,139 @@ +--- +title: "Rotations and Coverage" +description: "Combine on-call rotations with coverage windows, per-rotation timezones, and priority to build business-hours, follow-the-sun, and overnight coverage." +sidebar: + label: "Rotations & Coverage" + order: 3 +--- + +A single rotation covers the simple case. Real coverage often needs more: a daytime team alongside an after-hours rotation, a roster that follows working hours around the globe, or an overnight shift that runs past midnight. You build all of these by adding **more rotations** and giving each one the **coverage window** and **timezone** it should run in. + +:::note[Availability] +On-Call Schedules are a paid Incident Management feature. If you don't see **On-Call Schedules** in the Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## How overlapping rotations resolve + +By default, rotations are **additive**. At any moment, whoever's on call is the **union of every active rotation's current assignee(s)**, deduplicated by person. A rotation is *active* when the moment falls inside its coverage window; a 24/7 rotation is always active. + +So **where two rotations' coverage windows overlap, both responders are on call** — the building block for multi-responder coverage. When you want exactly one person on call there instead, you have two levers: give the rotations windows that don't overlap, or give one a higher **priority** so it takes over the overlap. Priority is covered [below](#rotation-priority). + +## Coverage windows + +Each rotation's **Coverage** is either always-on or limited to a window: + +- **24/7** — the rotation is always active. This is the right choice for a baseline rotation that an escalation policy can always reach. +- **Custom** — the rotation is active only within a window built from two parts: + - **Active days** — any set of weekdays, or a day range (for example Mon–Fri). + - **Active times** — **All day**, or a from–to window (for example 09:00–17:00). + +"All day" on the selected days means round-the-clock on those days. A from–to window applies on each active day. All of a rotation's times are evaluated in its **effective timezone** — its own timezone if set, otherwise the schedule's. + +A rotation exposes a **single** active-times window. If you need *different hours on different days* — say 09:00–17:00 on weekdays but all day on weekends — use **separate rotations**, one per pattern. + +![Rotation coverage settings](./images/rotation-coverage-settings.png) + +### Overnight windows that cross midnight + +An active-times window can end earlier than it starts — for example **22:00–05:00** — which means it runs overnight into the next morning. The rule to remember: + +> **Active days select the day a shift *starts*.** + +So a rotation with **Active days Mon–Fri** and **Active times 22:00–05:00** runs each weeknight from 22:00 into the next morning until 05:00: + +- It starts a shift on Monday, Tuesday, Wednesday, Thursday, and **Friday** nights. +- The Friday-night shift runs into **Saturday 05:00** — because it *started* on Friday. +- It does **not** start a shift on Saturday or Sunday night, so there's no coverage from Saturday 22:00 onward. + +If you also need Saturday and Sunday nights covered, add the missing days to the rotation (or add a second rotation for the weekend). This start-day rule is the most common point of confusion with overnight coverage, so check it on the calendar before you rely on it. + +## Rotation priority + +Trimming windows or adding days handles most overlaps. Sometimes two rotations *have* to cover the same moment, though, and you want only one of them on call. That's what **priority** does. + +Every rotation has a **priority** — a whole number, **0 by default**. Where rotations overlap in time, the **highest priority present wins**: it stays on call and hides every lower-priority rotation for as long as the overlap lasts. Rotations that share a priority stay additive, so they're all on call together. A higher number wins — a rotation at priority `1` takes over one left at `0`. + +Because every rotation starts at `0`, a schedule behaves exactly as it always has until you raise one. Priority does nothing until one rotation sits above another. You set it in the rotation drawer, right under the rotation name. + +### Hand an overlap to one rotation + +The common case is an overnight weekday rotation bleeding into the weekend. Pair the overnight after-hours rotation from above with an all-day weekend rotation: + +| Rotation | Coverage | Priority | +|---|---|---| +| After hours | Custom — Mon–Fri, 22:00–05:00 | 0 | +| Weekend | Custom — Sat–Sun, all day | 1 | + +The after-hours rotation's **Friday** shift starts on Friday and runs into **Saturday 05:00** — the start-day rule from above. That tail overlaps the Weekend rotation's Saturday. Left at the same priority, both people are on call from Saturday 00:00 to 05:00, so the weekday responder gets pulled in on a day off. + +Give **Weekend** the higher priority and the overlap resolves: the after-hours shift ends at **Saturday 00:00**, Weekend owns all of Saturday, and the calendar shows the handover at midnight. The rest of the week is untouched — the two rotations only interact where they overlap. + +:::tip[Leave priority at 0 to stay additive] +Default priority keeps every rotation additive — the same multi-responder and follow-the-sun behavior described on this page. Reach for priority only when an overlap should resolve to one rotation instead of both. +::: + +## Shift assignment within a rotation + +Each rotation also decides how its members share the shifts: + +- **Handoff and rotate to next member** — one member is on call per shift, and the duty advances to the next member at each handoff. +- **All members are on-call for each shift** — everyone in the rotation is on call together for as long as the rotation is active. Use this for a whole team that responds as a group. + +## Per-rotation timezone + +A rotation can override the schedule's timezone with its own. Set it, and that rotation's handoff and coverage times are evaluated in the rotation's timezone rather than the schedule's. Leave it on **Use schedule timezone** to inherit. This is what makes follow-the-sun work: each region expresses its hours in its *own* local time. + +## Example: business hours plus after hours + +Two additive rotations give you a daytime team alongside an always-available on-call rotation: + +| Rotation | Coverage | Members | +|---|---|---| +| Business hours | Custom — Mon–Fri, 09:00–17:00 | Day-shift team | +| After hours | 24/7 | On-call engineers | + +During business hours **both** rotations are active, so the day-shift responder *and* the on-call engineer are on call at once — an escalation step notifies both. Outside business hours only the 24/7 rotation is active, so the on-call engineer holds it alone. The always-on rotation guarantees there's never a moment with nobody on call. + +If you'd rather have exactly one person on call at a time, you have two options. Give the **Business hours** rotation a higher **priority** so it takes over during the day while the 24/7 rotation holds nights and weekends alone — one responder at every moment, no gaps. Or replace the 24/7 rotation with one whose window *complements* business hours (nights and weekends) so the two tile the week without overlapping, taking care not to leave a gap. (That second route usually takes two after-hours rotations, since a rotation has a single active-times window.) + +![Business-hours and after-hours rotations](./images/schedule-split-rotations.png) + +## Example: follow-the-sun + +To hand coverage around the globe, give each region its own rotation with its **own timezone** and a local active-times window. Each region works its own daytime, and coverage passes from one to the next as the day moves west: + +| Rotation (region) | Timezone | Active times (local) | +|---|---|---| +| APAC | Asia/Singapore | 09:00–17:00 | +| EMEA | Europe/London | 09:00–17:00 | +| Americas | America/New_York | 09:00–17:00 | + +Because each window is evaluated in the rotation's **own** timezone, you express every region's hours as plain local working hours — no timezone math. As the clock advances, coverage hands from APAC to EMEA to the Americas and back. Small overlaps between regions are fine: while they overlap, both regions' responders are on call — what you want at a handover. Keep every region at the same priority (the default `0`) so none masks another. + +:::tip[Each region's hours are its *own* local time] +Set each follow-the-sun rotation's timezone to the region it covers and enter 09:00–17:00 as that region's local hours. There's no need to translate everything into one shared timezone — that was a constraint of the old model, not this one. +::: + +## Example: overnight on-call + +A single rotation can carry an overnight shift. Using the 22:00–05:00 window from above: + +| Field | Value | +|---|---| +| Coverage | Custom | +| Active days | Mon–Fri | +| Active times | 22:00–05:00 | + +This covers each weeknight from 22:00 until 05:00 the next morning, including Friday night into Saturday at 05:00. It does not start a shift on Saturday or Sunday night — add those days if you need the weekend too. + +## Mind the gaps + +Custom coverage windows create uncovered time on purpose, but a schedule made entirely of windowed rotations can leave moments with **nobody on call**. If an incident fires during such a gap, an escalation step that targets the schedule has no one to reach. + +Make it a habit: for any schedule an escalation policy depends on, keep at least one **24/7 rotation**, or make sure your rotations' coverage windows tile the clock with no gap. The **Schedules** tab's coverage timeline (and the schedule's own calendar) is the quickest way to scan for an uncovered window before you rely on the schedule. + +## Related + +- [Create a schedule](../creating-a-schedule/) — rotation fields and the calendar editor. +- [Overrides](../overrides/) — one-off coverage on top of the rotations. +- [Escalation Policies](../../escalation-policy/) — what notifies a schedule. diff --git a/src/content/docs/docs/incident-management/routing-rules/_meta.json b/src/content/docs/docs/incident-management/routing-rules/_meta.json new file mode 100644 index 00000000..05394958 --- /dev/null +++ b/src/content/docs/docs/incident-management/routing-rules/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Routing Rules", + "order": 9 +} diff --git a/src/content/docs/docs/incident-management/routing-rules/images/rule-action-picker.png b/src/content/docs/docs/incident-management/routing-rules/images/rule-action-picker.png new file mode 100644 index 00000000..1342dec2 Binary files /dev/null and b/src/content/docs/docs/incident-management/routing-rules/images/rule-action-picker.png differ diff --git a/src/content/docs/docs/incident-management/routing-rules/images/rule-editor-conditions.png b/src/content/docs/docs/incident-management/routing-rules/images/rule-editor-conditions.png new file mode 100644 index 00000000..3679c3bf Binary files /dev/null and b/src/content/docs/docs/incident-management/routing-rules/images/rule-editor-conditions.png differ diff --git a/src/content/docs/docs/incident-management/routing-rules/images/rule-tester.png b/src/content/docs/docs/incident-management/routing-rules/images/rule-tester.png new file mode 100644 index 00000000..dda9f6f0 Binary files /dev/null and b/src/content/docs/docs/incident-management/routing-rules/images/rule-tester.png differ diff --git a/src/content/docs/docs/incident-management/routing-rules/index.mdx b/src/content/docs/docs/incident-management/routing-rules/index.mdx new file mode 100644 index 00000000..9c0a9af2 --- /dev/null +++ b/src/content/docs/docs/incident-management/routing-rules/index.mdx @@ -0,0 +1,128 @@ +--- +title: "Routing Rules" +description: "Decide how an incoming alert becomes an incident — its service, escalation policy, and severity — and whether to suppress it, add responders, or notify on-call." +sidebar: + label: "Overview" + order: 1 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Routing rules decide what happens to an alert the moment it reaches Incident Management: which **service** the incident opens against, which **escalation policy** pages, what **severity** it gets, and whether to suppress it, add responders, or notify an on-call schedule. + +:::caution[Two features share this name] +This page is about **incident routing inside IM** — turning an incoming alert into the right incident. It is *not* the same as [**Alerts → Routing Rules**](../../alerts/routing-rules/), which routes alert *notifications* to channels like Slack. If you're trying to control which Slack channel gets pinged, you want that page instead. +::: + +:::note[Availability] +Routing Rules are a paid Incident Management feature. If you don't see **Routing Rules** in the Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## What a routing rule decides + +When an alert arrives from an [alert source](../integrations/), KloudMate runs it through your routing rules in priority order. A rule that matches can: + +- **Assign a service** — open the incident against a specific service. +- **Assign an escalation policy** — choose which policy pages. +- **Set the severity** — override the severity carried by the alert. +- **Suppress the incident** — drop the alert so no incident opens. +- **Add responders** — attach specific users to the incident. +- **Notify on-call schedules** — notify whoever is on call on one or more [schedules](../oncall/) — possibly several people per schedule. + +## Scope, priority, and order + +Each rule has three controls that decide when it runs: + +- **Scope** — a rule is either **workspace-wide** (applies to every alert) or scoped to a **single service**. When two rules share a priority, service-scoped rules are evaluated ahead of workspace-wide ones. +- **Priority** — a number where **lower wins**. Rules run from lowest priority upward. Give specific rules low numbers and broad catch-alls high ones. New rules default to `100`. +- **Enabled** — a toggle. Disabled rules are skipped entirely, which lets you stage a rule before turning it on. + +## Match conditions + +A rule's matcher has one of three modes: + +- **All events** — the rule matches every alert, with no conditions. +- **All conditions** — every condition must be true (AND). +- **Any condition** — at least one condition must be true (OR). + +Each condition picks a **field**, an **operator**, and a **value**. The available operators depend on the field: + +| Field | What it matches | Operators | +|---|---|---| +| Severity | The severity carried by the alert | Equals, Not equals, In | +| Service | The target service | Equals, Not equals, In | +| Alert Source | The integration the alert came from | Equals, Not equals, In | +| Tag | A tag on the payload, addressed as `tag.<key>` (for example `tag.environment`) | Equals, Not equals, Contains, Matches regex, In | +| Payload Field | Any field in the raw payload, addressed as `payload.<path>` (for example `payload.details.region`) | Equals, Not equals, Contains, Matches regex, In, Greater than, Less than | + +**In** matches against a comma-separated set of values. **Matches regex** tests the value against a regular expression. **Greater than** / **Less than** compare numerically, so they're available only on payload fields. + +![Routing rule editor with match conditions](./images/rule-editor-conditions.png) + +## Actions + +When a rule matches, its actions feed a single routing decision. How conflicting rules combine depends on the action: + +| Action | What it does | When rules conflict | +|---|---|---| +| Assign service | Routes the incident to a service | **First match wins** | +| Assign escalation policy | Sets which policy pages | **First match wins** | +| Set severity | Overrides the incident's severity | **First match wins** | +| Suppress incident | Drops the alert — no incident opens | **First match wins**, and stops evaluation | +| Add responders | Attaches users to the incident | **Accumulates** across every matching rule | +| Notify on-call schedules | Notifies the people on call on the named schedules | **Accumulates** across every matching rule | + +"First match wins" means the highest-priority rule to set a value owns it; later rules can't override it. "Accumulates" means responders and notified schedules from *every* matching rule are merged together (duplicates removed). + +:::caution[Suppress stops everything] +A rule that suppresses the incident ends evaluation immediately — no lower-priority rule runs, and no responders or on-call notifications from other rules are added. Put suppression rules at a low priority and scope them tightly. +::: + +![Routing rule action picker](./images/rule-action-picker.png) + +## How service, policy, and severity are resolved + +A rule action is only the first place KloudMate looks. For each of these fields it takes the first value it finds: + +- **Escalation policy** — the rule's action, then the alert source's default policy, then the service's default policy. +- **Severity** — the rule's action, otherwise the severity mapped from the incoming alert. +- **Service** — the rule's action, otherwise the service the alert source is wired to. + +So a rule doesn't have to set everything. If you only want to bump severity for a subset of alerts, set that one action — service and policy still resolve from the alert source and service defaults. + +## Test a rule before saving + +The rule editor includes a tester so you don't have to wait for a real alert. Feed it a sample **severity**, **service**, and **payload** (as JSON), and it shows the decision the rules would produce: + +- which rules matched, +- the final service, escalation policy, and severity, +- whether the incident would be suppressed, +- the responders and on-call schedules that would be notified. + +Use it to confirm a new rule does what you expect — and that it doesn't accidentally suppress traffic you care about — before it goes live. + +![Routing rule tester with a decision result](./images/rule-tester.png) + +## Create a routing rule + +<Steps> +1. Open **Incident Management → Routing Rules** and click **Create rule**. + +2. Name the rule, set its **scope** (workspace-wide or a single service), and set a **priority** (lower wins). + +3. Choose a **match mode** — All events, All conditions, or Any condition — and add conditions. + +4. Add **actions**: assign a service or escalation policy, set severity, suppress, add responders, or notify on-call schedules. + +5. **Test** the rule against a sample payload and confirm the decision. + +6. **Save**. The rule lands at its priority, enabled by default. +</Steps> + +## Related + +- [Alert Sources](../integrations/) — where alerts enter IM. +- [Services](../services/) — the routing target and its default escalation policy. +- [Escalation Policies](../escalation-policy/) — what a rule assigns. +- [On-Call Schedules](../oncall/) — what "Notify on-call schedules" pages. +- [Alerts → Routing Rules](../../alerts/routing-rules/) — the different feature that routes alert notifications to channels. diff --git a/src/content/docs/docs/incident-management/services/creating-services.mdx b/src/content/docs/docs/incident-management/services/creating-services.mdx index ac6e606c..5ba074b5 100644 --- a/src/content/docs/docs/incident-management/services/creating-services.mdx +++ b/src/content/docs/docs/incident-management/services/creating-services.mdx @@ -1,27 +1,27 @@ --- title: "Create a Service" -description: "Create a service and define its owner, default escalation policy, and SLA target." +description: "Create a service and define its owner and default escalation policy." sidebar: label: "Create a Service" order: 2 --- +import { Steps } from '@astrojs/starlight/components'; + Use the steps below to create a new service. +<Steps> 1. Click **Create Service** from the Services page. -![image](./images/creating-services-1.jpeg) +2. Enter the **service name** and a **description**. Both are required. -2. Enter the **Service Name** and **Description**. -3. Select the **Service Owner**. -4. Select the default **Escalation Policy** for the service. +3. Select the **service owner** — a user in your workspace. -:::note - Every service requires an escalation policy. If one does not already exist, create it first before saving the service. -::: +4. Select the default **escalation policy**. Every service requires one. If you don't have a policy yet, use the **Create new escalation policy** link in the form to make one without leaving the page. -5. Set the **SLA Target** percentage. +5. Click **Save**. +</Steps> ![image](./images/creating-services-1.png) -6. Click **Save**. +Once the service exists, open it to add [alert sources](../../integrations/) and review its incidents. diff --git a/src/content/docs/docs/incident-management/services/images/creating-services-1.png b/src/content/docs/docs/incident-management/services/images/creating-services-1.png index b45ae96c..e0c06d45 100644 Binary files a/src/content/docs/docs/incident-management/services/images/creating-services-1.png and b/src/content/docs/docs/incident-management/services/images/creating-services-1.png differ diff --git a/src/content/docs/docs/incident-management/services/images/service-detail.png b/src/content/docs/docs/incident-management/services/images/service-detail.png new file mode 100644 index 00000000..2abdc618 Binary files /dev/null and b/src/content/docs/docs/incident-management/services/images/service-detail.png differ diff --git a/src/content/docs/docs/incident-management/services/images/services-1.jpeg b/src/content/docs/docs/incident-management/services/images/services-1.jpeg index 66484088..ca153349 100644 Binary files a/src/content/docs/docs/incident-management/services/images/services-1.jpeg and b/src/content/docs/docs/incident-management/services/images/services-1.jpeg differ diff --git a/src/content/docs/docs/incident-management/services/index.mdx b/src/content/docs/docs/incident-management/services/index.mdx index cfbeeaaa..17f80aae 100644 --- a/src/content/docs/docs/incident-management/services/index.mdx +++ b/src/content/docs/docs/incident-management/services/index.mdx @@ -1,41 +1,38 @@ --- title: "Services" -description: "Group incidents by service and manage related integrations and escalation defaults." +description: "Group incidents by service, and give each an owner and a default escalation policy." sidebar: label: "Overview" order: 1 --- -Services group incidents by application area, microservice, or operational ownership so your team can manage related incidents together. +Services group incidents by application area, microservice, or operational ownership, so your team manages related incidents together. Each service has an **owner** and a default **escalation policy**. -![image](./images/services-1.jpeg) +To track a service against a reliability target, define a [Service Level Objective (SLO)](../../reliability/) for it. -The Services page lists each service with its: +![image](./images/services-1.jpeg) -- Name -- Description -- Owner -- Default escalation policy -- Current health state +The Services page lists each service with its name, description, owner, default escalation policy, and current health. :::note - A service can have multiple integrations. The escalation policy shown on the service is the default policy used when an integration does not define its own override. +A service can own several alert sources. The escalation policy on the service is the default — an alert source uses it unless it sets its own override, and a [routing rule](../routing-rules/) can override it per incident. ::: -## Service Details +## Service details -Open any service to view: +Open a service to review: - Triggered, acknowledged, and resolved incident counts -- Service owner details -- Related integrations - -![image](./images/services-1.png) +- The service owner +- The default escalation policy (linked) +- The service's **alert sources**, listed with their name, description, type, and escalation policy -You can also add an integration directly from the service page. +You can add an alert source directly from the service page, and use the settings menu to edit or archive the service. -![image](./images/services-2.jpeg) +![Service detail with alert sources](./images/service-detail.png) -Use the settings menu to edit or archive the service. +## Next -![image](./images/services-3.jpeg) +- [Create a service](./creating-services/) — set its owner and escalation policy. +- [Alert Sources](../integrations/) — what feeds incidents into a service. +- [SLOs](../../reliability/) — track a service against a reliability target. diff --git a/src/content/docs/docs/incident-management/slack-integration.mdx b/src/content/docs/docs/incident-management/slack-integration.mdx index 928659fb..b6cf81e4 100644 --- a/src/content/docs/docs/incident-management/slack-integration.mdx +++ b/src/content/docs/docs/incident-management/slack-integration.mdx @@ -1,25 +1,33 @@ --- title: "Slack Integration" -description: "Connect Slack to Incident Management so responders can receive and manage incidents through Slack." +description: "Connect Slack to Incident Management so responders can receive and manage incidents from Slack." sidebar: label: "Slack Integration" - order: 8 + order: 11 --- -Use the Slack integration to send incident notifications into Slack and let responders interact with incidents from their workspace. +KloudMate uses a single unified Slack app for both notification delivery and IM ChatOps. Installing it once gives you: -## Connect Slack +- Threaded notifications on alert groups and incidents (replies stay attached to the parent message). +- Per-user ChatOps actions — acknowledge, resolve, comment on incidents without leaving Slack. +- Cross-surface visibility — the same install shows up under both **Settings → Notification Channels** and **Incidents → Settings → ChatOps**. -1. Open **Incidents -> Settings** from the left navigation. +You can start the install from either entry point — the OAuth flow lands the same row in both surfaces. -![image](./images/slack-integration-1.png) +## Connect Slack from Incident Management -2. In the **Chat Ops** tab, click **Connect** for Slack. -3. Complete the Slack authorization flow. -4. Select the Slack channel you want to use. -5. Add a message for your app manager if required. -6. Submit the connection. +1. Open **Incidents → Settings** from the left navigation. -:::note - After the workspace connection is complete, each team member must connect their own Slack account before they can manage incidents from Slack. -::: +![Slack Chat Ops settings](./images/slack-integration-1.png) + +2. In the **Chat Ops** tab, click **Connect** under the Slack card. Only workspace admins can run this step. +3. Complete the Slack OAuth flow — pick the Slack workspace and channel KloudMate should post into. +4. KloudMate returns you to the Chat Ops tab and shows the linked workspace + channel. + +The new Slack row appears under both **Settings → Notification Channels** and **Incidents → Settings → Chat Ops**. You can use it as a notification destination on a [Routing Rule](../../alerts/routing-rules/) immediately. + +## Per-user account linking + +After the org-level install is complete, each responder needs to link their own Slack account before they can acknowledge or resolve incidents from Slack. From the Chat Ops tab, scroll to **User Settings** and click **Connect account** — a one-time OAuth flow links your KloudMate identity to your Slack account. + +Once linked, the same tab shows your Slack username with a **Disconnect** action. diff --git a/src/content/docs/docs/incident-management/status-pages/_meta.json b/src/content/docs/docs/incident-management/status-pages/_meta.json new file mode 100644 index 00000000..9dadcba8 --- /dev/null +++ b/src/content/docs/docs/incident-management/status-pages/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Status Pages", + "order": 10 +} diff --git a/src/content/docs/docs/incident-management/status-pages/branding-and-custom-domain.mdx b/src/content/docs/docs/incident-management/status-pages/branding-and-custom-domain.mdx new file mode 100644 index 00000000..e3f0f3d3 --- /dev/null +++ b/src/content/docs/docs/incident-management/status-pages/branding-and-custom-domain.mdx @@ -0,0 +1,59 @@ +--- +title: "Branding and Custom Domain" +description: "Add your logo, colors, and favicon, and serve the status page from your own domain with automatic TLS." +sidebar: + label: "Branding and Custom Domain" + order: 3 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Branding makes the status page look like yours rather than a generic KloudMate page; a custom domain puts it on a URL your customers already trust. Both live under the editor's **Settings** tab. + +## Branding + +Set these in **Settings → Branding**: + +- **Logo** — PNG, JPEG, or WebP, up to **2 MB**. KloudMate downscales it to a 512px longest edge. +- **Favicon** — PNG, JPEG, or WebP, up to **1 MB**, square (1:1). It's downscaled to 256px. +- **Primary color** — the page's accent color. +- **Hide "Powered by KloudMate"** — removes the KloudMate footer badge. This is a paid add-on; you'll be prompted to upgrade if your plan doesn't include it. + +When you upload a logo or favicon, a crop dialog lets you frame it before it's saved. + +![Status page branding settings](./images/status-page-branding.png) + +## The default URL + +Every status page is reachable for free at a KloudMate-hosted address built from its slug: + +``` +https://<slug>.<your-status-domain> +``` + +This works the moment you publish — no DNS setup required. Use it as-is, or move to a custom domain. + +## Custom domain + +A custom domain serves the page from a subdomain of your own — for example `status.example.com`. Custom domains are a paid feature, and only an **organization owner** can set one up. + +<Steps> +1. In **Settings**, open the custom domain panel and enter your **subdomain** (for example `status.example.com`). It has to be a subdomain — an apex domain like `example.com` isn't supported. + +2. KloudMate returns two DNS records: a **CNAME** and a **TXT** record. + +3. Add both records at your DNS provider. + +4. Come back and click **Verify**. The domain's status moves to **pending** while KloudMate checks the records. + +5. Once the records resolve, the status flips to **active** and TLS is provisioned automatically. Your page is now live on the custom domain. +</Steps> + +If verification doesn't go through, the status shows **failed** — double-check that both records match exactly and that DNS has had time to propagate, then verify again. + +![Custom domain DNS records and verification](./images/status-page-custom-domain.png) + +## Related + +- [Status Pages overview](../) — components and publishing. +- [Post incidents and updates](../posting-incidents/) — what visitors read on the page. diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-page-branding.png b/src/content/docs/docs/incident-management/status-pages/images/status-page-branding.png new file mode 100644 index 00000000..74e962b5 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-page-branding.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-page-components-tab.png b/src/content/docs/docs/incident-management/status-pages/images/status-page-components-tab.png new file mode 100644 index 00000000..916ea0d6 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-page-components-tab.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-page-custom-domain.png b/src/content/docs/docs/incident-management/status-pages/images/status-page-custom-domain.png new file mode 100644 index 00000000..0c9cea68 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-page-custom-domain.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-page-incident-composer.png b/src/content/docs/docs/incident-management/status-pages/images/status-page-incident-composer.png new file mode 100644 index 00000000..dcc9b158 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-page-incident-composer.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-page-public.png b/src/content/docs/docs/incident-management/status-pages/images/status-page-public.png new file mode 100644 index 00000000..8d72f526 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-page-public.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/images/status-pages-list.png b/src/content/docs/docs/incident-management/status-pages/images/status-pages-list.png new file mode 100644 index 00000000..99cbbe68 Binary files /dev/null and b/src/content/docs/docs/incident-management/status-pages/images/status-pages-list.png differ diff --git a/src/content/docs/docs/incident-management/status-pages/index.mdx b/src/content/docs/docs/incident-management/status-pages/index.mdx new file mode 100644 index 00000000..3b40c2bf --- /dev/null +++ b/src/content/docs/docs/incident-management/status-pages/index.mdx @@ -0,0 +1,75 @@ +--- +title: "Status Pages" +description: "Publish a branded status page so customers can see component health, active incidents, and 90-day uptime history." +sidebar: + label: "Overview" + order: 1 +--- + +import { Steps } from '@astrojs/starlight/components'; + +A status page tells your customers what's working and what isn't, without them opening a ticket. You publish component health, active incidents, and uptime history at a public URL; when something breaks, you post updates there instead of answering the same question a hundred times. + +:::note[Availability] +Status Pages are a paid Incident Management feature. If you don't see **Status Pages** in the Incident Management navigation, your plan doesn't include it yet — contact your account team to turn it on. +::: + +## What visitors see + +A published status page shows: + +- An **overall status banner** — Operational, Degraded Performance, Partial Outage, or Major Outage — derived from active incidents and component health. +- **Components**, optionally grouped, each with a **90-day uptime strip**. Hovering a day shows its status, duration, and any related incidents. +- **Active incidents**, newest update first. +- A **history** view for browsing past months. +- **RSS and Atom feeds** so people can subscribe to changes. + +![Public status page](./images/status-page-public.png) + +## The editor + +You manage everything from one editor with three tabs: + +- **Components** — what the page reports on. +- **Incidents** — what you post during an outage. See [Post incidents and updates](./posting-incidents/). +- **Settings** — branding, the public URL, and a custom domain. See [Branding and custom domain](./branding-and-custom-domain/). + +## Create a status page + +<Steps> +1. Open **Incident Management → Status Pages** and click **Create**. + +2. Enter a **name** — the title visitors see. + +3. Set a **slug**. It becomes part of the default public URL, so use lowercase letters, numbers, and hyphens (for example `acme-status`). + +4. Add an optional **description** shown under the title. + +5. Pick a **primary color** for the page's accent. The default is `#1453ff`. + +6. Save. The page starts as a **draft** — only you can see it until you publish. +</Steps> + +## Add components + +Components are the services or features your page reports on — "API", "Dashboard", "Webhooks". For each component you set: + +- **Name** — what visitors see. +- **Group** *(optional)* — components with the same group label are visually grouped on the page. +- **Linked service** *(optional)* — link a component to an [IM service](../services/) and its status tracks that service's health automatically, so you don't update it by hand. +- **Display order** — reorder components to control how they stack on the page. + +A component's status is one of **Operational**, **Degraded**, **Partial Outage**, or **Major Outage**. (On the public page, "Degraded" reads as "Degraded Performance".) + +![Status page editor Components tab](./images/status-page-components-tab.png) + +## Publish and unpublish + +A page is either a **draft** (only your team can see it) or **public**. Publish when the components are set up and you're ready for visitors; unpublish to take it private again. Publishing doesn't lock anything — you keep editing components, incidents, and settings while it's live. + +![Status Pages list](./images/status-pages-list.png) + +## In this section + +- [Post incidents and updates](./posting-incidents/) — communicate during an outage and publish from an existing incident. +- [Branding and custom domain](./branding-and-custom-domain/) — logo, colors, and serving the page from your own domain. diff --git a/src/content/docs/docs/incident-management/status-pages/posting-incidents.mdx b/src/content/docs/docs/incident-management/status-pages/posting-incidents.mdx new file mode 100644 index 00000000..f6e3bc2c --- /dev/null +++ b/src/content/docs/docs/incident-management/status-pages/posting-incidents.mdx @@ -0,0 +1,56 @@ +--- +title: "Post Incidents and Updates" +description: "Communicate during an outage: create a status-page incident, post updates over time, and publish straight from an IM incident." +sidebar: + label: "Post Incidents and Updates" + order: 2 +--- + +import { Steps } from '@astrojs/starlight/components'; + +When something breaks, a status-page incident is how you tell customers what's happening and keep them posted as you work. You can write one from scratch in the editor, or publish an existing Incident Management incident to the page in a couple of clicks. + +## Create a status-page incident + +<Steps> +1. In the status page editor, open the **Incidents** tab and start a new incident. + +2. Write a **title** — what customers will see at the top of the page. + +3. Set the **status**: **Investigating**, **Identified**, **Monitoring**, or **Resolved**. A new incident usually starts at *Investigating*. + +4. Set the **impact**: **None**, **Minor**, **Major**, or **Critical**. Impact drives how the incident shows on the overall status banner — minor shows as degraded performance, major as a partial outage, critical as a major outage. + +5. Choose the **affected components**. Their status reflects the incident while it's open. + +6. Decide whether the incident is **public** now or hidden. Turn public off to draft it — line up the first update before anyone sees it — then make it public when you're ready. +</Steps> + +## Post updates as you learn more + +An incident is a running log, not a single message. As you investigate, add **updates** — each has its own body and its own status — and the page shows them newest first. A typical arc: + +1. **Investigating** — "We're looking into elevated error rates on the API." +2. **Identified** — "A bad deploy is the cause. Rolling back now." +3. **Monitoring** — "The rollback is complete. Watching error rates." +4. **Resolved** — "Error rates are back to normal. The incident is resolved." + +Each update moves the incident's status forward, so visitors see both the current state and the timeline that got there. + +![Status page incident with update composer](./images/status-page-incident-composer.png) + +## Publish from an Incident Management incident + +When you're already responding to an incident in IM, you don't have to retype it. Open the incident and use **Publish to status page**. + +The publish dialog pre-fills what it can: it picks a status page linked to the incident's service, pre-selects that page's linked components, and carries over the incident's title. Adjust the impact and components, choose whether to publish publicly or as a draft, and post. + +This action is **idempotent** — publishing the same incident again updates the status-page incident it already created rather than making a duplicate. So as the IM incident progresses, you can re-publish to keep the customer-facing version in sync. + +See [Incident details](../../incidents/incident-details/) for where the button lives. + +## Related + +- [Status Pages overview](../) — components and publishing. +- [Branding and custom domain](../branding-and-custom-domain/) — make the page yours. +- [Incident details](../../incidents/incident-details/) — the "Publish to status page" action. diff --git a/src/content/docs/docs/incident-management/what-is-incident-management.mdx b/src/content/docs/docs/incident-management/what-is-incident-management.mdx index f0f06531..57e5513d 100644 --- a/src/content/docs/docs/incident-management/what-is-incident-management.mdx +++ b/src/content/docs/docs/incident-management/what-is-incident-management.mdx @@ -5,48 +5,76 @@ sidebar: label: "What Is Incident Management?" order: 2 --- -An incident is any event or issue that negatively affects services, application workflows, end users, or customers. +An incident is any event or issue that negatively affects your services, application workflows, end users, or customers. -KloudMate Incident Management helps teams detect, assign, escalate, and resolve incidents within a unified workflow. +KloudMate Incident Management gives your team one place to detect, route, assign, escalate, and resolve those incidents — from the moment an alert fires to the update you post for customers. -## What the Module Covers +## What the module covers -Incident Management in KloudMate helps your team: +Incident Management helps your team: -- Detect and track active incidents -- Assign incidents to the right service and responders -- Route alerts through integrations -- Apply escalation rules automatically -- Collaborate on investigation and resolution +- Detect and track active incidents in one view. +- Route incoming alerts to the right service, severity, and escalation policy automatically. +- Notify the people on call through on-call schedules. +- Escalate step by step until someone acknowledges. +- Keep customers informed with a public status page. -## Key Components +## How a signal becomes a resolved incident + +A monitored condition trips an alert, and from there it follows a predictable path: + +1. An **alert source** receives the alert — from KloudMate alerts, CloudWatch, or a generic webhook. +2. A **routing rule** decides which service the incident opens against, which escalation policy pages, and what severity it carries. A rule can also suppress noise before it becomes an incident at all. +3. The **incident** opens on its **service**. +4. The service's **escalation policy** notifies responders step by step until someone acknowledges. +5. When a step targets an **on-call schedule**, KloudMate resolves it to whoever is on call at that moment — possibly several people — and notifies each of them. +6. Once you're responding, you can **publish the incident to a status page** to keep customers in the loop. + +Routing rules and status pages are optional, but this is the backbone every incident moves along. + +## Key components ![image](./images/what-is-incident-management_-1.jpeg) ### Overview -The overview page gives you a real-time snapshot of incident counts, statuses, MTTA, and MTTR. +A real-time snapshot of incident counts, statuses, MTTA, and MTTR. ### Incidents -Incidents are the core records your team acknowledges, investigates, and resolves. +The core records your team acknowledges, investigates, and resolves. Each one carries an activity timeline of every status change and notification. See [Incidents](../incidents/). ### Services -Services group incidents by application area, microservice, or functional ownership. +Services group incidents by application area or ownership. Each carries an owner and a default escalation policy. See [Services](../services/). -### Integrations +### Alert Sources -Integrations receive alerts from KloudMate alarms, CloudWatch, webhooks, and other sources. +Alert sources receive alerts from KloudMate, CloudWatch, and webhooks, then map each incoming payload to an incident. (Formerly called Integrations.) See [Alert Sources](../integrations/). ### Escalation Policies -Escalation policies define who gets notified, how they are notified, and when incidents move to the next response step. +Escalation policies define who gets notified, how, and when an unacknowledged incident moves to the next step. Steps can target users, on-call schedules, and Slack channels. See [Escalation Policies](../escalation-policy/). + +### On-Call Schedules + +Additive rotations that decide who's on call at any moment, with coverage windows and one-off overrides. A schedule can resolve to several people at once. Escalation policies and routing rules resolve a schedule to whoever is on call at notification time. See [On-Call Schedules](../oncall/). + +### Routing Rules + +Rules that decide, when an alert reaches IM, which service and escalation policy an incident gets, its severity, and whether to suppress it. See [Routing Rules](../routing-rules/). + +### Status Pages + +Branded public pages that show component health, active incidents, and uptime history, so customers can self-serve during an outage. See [Status Pages](../status-pages/). ## Related Resources - [Incident Lifecycle](../incident-lifecycle/) - [Incidents](../incidents/) - [Services](../services/) -- [Integrations](../integrations/) +- [Alert Sources](../integrations/) - [Escalation Policies](../escalation-policy/) +- [On-Call Schedules](../oncall/) +- [Routing Rules](../routing-rules/) +- [Status Pages](../status-pages/) diff --git a/src/content/docs/docs/infrastructure/index.mdx b/src/content/docs/docs/infrastructure/index.mdx index 35e9c5ab..b6eafeb8 100644 --- a/src/content/docs/docs/infrastructure/index.mdx +++ b/src/content/docs/docs/infrastructure/index.mdx @@ -3,7 +3,7 @@ title: "Infrastructure Monitoring" description: "Collect and monitor infrastructure metrics from servers, Kubernetes, Prometheus, vCenter, and cloud environments in KloudMate." sidebar: label: "Overview" - order: 1 + order: 0 --- Infrastructure monitoring in KloudMate gives you visibility into the health, performance, and capacity of the systems your applications run on. @@ -35,4 +35,4 @@ Once metrics begin flowing, use: - [Explore](../visualize-data/explore/) for ad hoc investigation - [Dashboards](../../visualize-data/dashboards/) for curated visualizations -- [Alerts](../../alarms/) to notify on infrastructure health changes +- [Alerts](../alerts/) to notify on infrastructure health changes diff --git a/src/content/docs/docs/infrastructure/integrations/prometheus-integration.mdx b/src/content/docs/docs/infrastructure/integrations/prometheus-integration.mdx index 2e1f77ee..e94c6fc1 100644 --- a/src/content/docs/docs/infrastructure/integrations/prometheus-integration.mdx +++ b/src/content/docs/docs/infrastructure/integrations/prometheus-integration.mdx @@ -69,7 +69,7 @@ After the Agent restarts: - open [Explore](../../../visualize-data/explore/) - query one of the scraped Prometheus metrics -- build dashboards or alarms once the data is visible +- build dashboards or alerts once the data is visible ## Related Paths diff --git a/src/content/docs/docs/infrastructure/kubernetes/index.mdx b/src/content/docs/docs/infrastructure/kubernetes/index.mdx index 8621b9a9..da503902 100644 --- a/src/content/docs/docs/infrastructure/kubernetes/index.mdx +++ b/src/content/docs/docs/infrastructure/kubernetes/index.mdx @@ -37,7 +37,7 @@ Once data is flowing, use: - [Explore](../../../visualize-data/explore/) for ad hoc queries - [Log Explorer](../../../logs/log-explorer/) to investigate pod logs - [Dashboards](../../../visualize-data/dashboards/) for curated visualizations -- [Alerts](../../../alarms/) to notify on resource or availability issues +- [Alerts](../../alerts/) to notify on resource or availability issues ## Unified Monitoring for Apps and Databases diff --git a/src/content/docs/docs/issues.mdx b/src/content/docs/docs/issues.mdx deleted file mode 100644 index 1b01f721..00000000 --- a/src/content/docs/docs/issues.mdx +++ /dev/null @@ -1,81 +0,0 @@ ---- -title: "Issues" -description: "Learn how KloudMate automatically discovers, aggregates, and helps you resolve errors and warnings across your services." -sidebar: - order: 17 ---- -KloudMate automatically discovers errors from your application logs and traces and aggregates them in the Issues section. This gives you a centralised view of all errors and warnings detected across your services, so you can track, investigate, and resolve them efficiently. - -To navigate to the Issues screen, click **Issues** in the left navigation menu. - -## Issues Dashboard - -The **Issues** screen displays all detected issues in a list. - -![image](./images/issues/issues-1.png) - -Each issue shows the source, last seen time, occurrence count, and assigned user. - -Use the **Assignee** and **Service** filters to narrow down the list. - -Issues are organised across three tabs; **All Unresolved** , **Ignored** , and **Resolved**, so you can focus on what’s active or review previously handled issues. - -Click the **Explore Error Patterns** button at the top right to identify recurring error patterns across your services. - -![image](./images/issues/issues-1.jpeg) - -## Issue Details - -Click on any issue to open its detail view. - -![image](./images/issues/issues-2.png) - -This screen shows the service it belongs to, the source, when it was first and last seen, and the total occurrence count. An occurrence frequency chart gives you a visual timeline of how often the issue has been happening over time. - -The **Related Errors** section lists individual occurrences of the issue with timestamps, service names, and direct links to the associated logs for deeper investigation. - -From this screen you can: - -- **Assign** the issue to a team member using the assign icon. - -![image](./images/issues/issues-2.jpeg) - -- **Add comments** using the comment icon: you can mention other users in comments, and all participating users receive notifications for any updates on the issue. - -![image](./images/issues/issues-3.jpeg) - -- **Mark as Resolved:** removes the issue from the Unresolved list. -- **Ignore:** mutes future notifications for this issue if it reoccurs. - -![image](./images/issues/issues-4.jpeg) - -- **Investigate with KloudMate Assistant:** automatically analyses the issue and provides a concise summary, probable root cause, and recommended actions for faster resolution. - -Issues can also be assigned to multiple users. Assigned users are notified when an issue is assigned to them, and all participating users receive updates for any subsequent activity on the issue. - -## Issue Statuses - -- **Unresolved:** An issue that has been detected and is yet to be reviewed or addressed. -- **Resolved:** An issue that has been reviewed and marked as resolved. It will no longer appear in the Unresolved list. -- **Ignored:** An issue that has been intentionally muted. No notifications will be sent if the same issue reoccurs in the future. - -You can mark issues as resolved or ignored either from the issue details page, or by selecting multiple issues from the main list and applying the action collectively. - -## Notification Preferences - -Users can configure a notification policy to receive alerts for: - -- **New Issues:** Issues detected for the first time. -- **New and Regression Issues:** Issues detected for the first time, as well as issues that had previously occurred and have reappeared. - -## User Permissions - -Access to issue actions is determined by the user’s assigned role: - -- **Owner and Developer:** Can mark issues as resolved or ignored, assign issues to team members, and leave comments. -- **Viewer:** Can assign issues to team members and leave comments, but cannot mark issues as resolved or ignored. - -## Related Resources - -- [Setting up Notification Policies](../alarms/notifications/) -- [Setting up Notification Channels](../platform/settings/notification-channels/) diff --git a/src/content/docs/docs/issues/_meta.json b/src/content/docs/docs/issues/_meta.json new file mode 100644 index 00000000..00b2f994 --- /dev/null +++ b/src/content/docs/docs/issues/_meta.json @@ -0,0 +1,5 @@ +{ + "label": "Issues", + "order": 17, + "collapsed": true +} diff --git a/src/content/docs/docs/issues/images/issues-1.jpeg b/src/content/docs/docs/issues/images/issues-1.jpeg new file mode 100644 index 00000000..5e4c3b60 Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-1.jpeg differ diff --git a/src/content/docs/docs/issues/images/issues-1.png b/src/content/docs/docs/issues/images/issues-1.png new file mode 100644 index 00000000..932f59c2 Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-1.png differ diff --git a/src/content/docs/docs/images/issues/issues-2.jpeg b/src/content/docs/docs/issues/images/issues-2.jpeg similarity index 100% rename from src/content/docs/docs/images/issues/issues-2.jpeg rename to src/content/docs/docs/issues/images/issues-2.jpeg diff --git a/src/content/docs/docs/issues/images/issues-2.png b/src/content/docs/docs/issues/images/issues-2.png new file mode 100644 index 00000000..92b9a4fe Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-2.png differ diff --git a/src/content/docs/docs/images/issues/issues-3.jpeg b/src/content/docs/docs/issues/images/issues-3.jpeg similarity index 100% rename from src/content/docs/docs/images/issues/issues-3.jpeg rename to src/content/docs/docs/issues/images/issues-3.jpeg diff --git a/src/content/docs/docs/issues/images/issues-3.png b/src/content/docs/docs/issues/images/issues-3.png new file mode 100644 index 00000000..6ae049de Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-3.png differ diff --git a/src/content/docs/docs/images/issues/issues-4.jpeg b/src/content/docs/docs/issues/images/issues-4.jpeg similarity index 100% rename from src/content/docs/docs/images/issues/issues-4.jpeg rename to src/content/docs/docs/issues/images/issues-4.jpeg diff --git a/src/content/docs/docs/issues/images/issues-4.png b/src/content/docs/docs/issues/images/issues-4.png new file mode 100644 index 00000000..e5a2f0ac Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-4.png differ diff --git a/src/content/docs/docs/issues/images/issues-notification-settings.png b/src/content/docs/docs/issues/images/issues-notification-settings.png new file mode 100644 index 00000000..68a5d7ca Binary files /dev/null and b/src/content/docs/docs/issues/images/issues-notification-settings.png differ diff --git a/src/content/docs/docs/issues/index.mdx b/src/content/docs/docs/issues/index.mdx new file mode 100644 index 00000000..26247084 --- /dev/null +++ b/src/content/docs/docs/issues/index.mdx @@ -0,0 +1,96 @@ +--- +title: Issues +description: Learn how KloudMate automatically discovers, aggregates, and helps + you resolve errors and warnings across your services. +editUrl: true +head: [] +template: doc +sidebar: + order: 1 + label: Overview + hidden: false + attrs: {} +pagefind: true +draft: false +--- + +KloudMate automatically discovers errors from your application logs and traces and aggregates them in the Issues section. This gives you a centralised view of all errors and warnings detected across your services, so you can track, investigate, and resolve them efficiently. + +To navigate to the Issues screen, click **Issues** in the left navigation menu. + +## Issues Dashboard + +The **Issues** screen displays all detected issues in a list. + +![Issues dashboard showing the list of all unresolved issues](./images/issues-1.png) + +Each issue shows the source, last seen time, occurrence count, and assigned user. + +Use the **Assignee**, **Service**, **Source**, and **Error type** filters to narrow down the list. + +Issues are organised across three tabs — **All Unresolved**, **Ignored**, and **Resolved** — so you can focus on what's active or review previously handled issues. + +Click **Notification settings** at the top right to configure when you receive notifications for new or recurring issues. + +Click **Explore Error Patterns** at the top right to identify recurring error patterns across your services. + +## Issue Details + +Click on any issue to open its detail view. + +![Issue detail page showing error message, metadata, occurrence chart, stacktrace, and infrastructure context](./images/issues-2.png) + +The detail page shows: + +- The full **error message** at the top +- **Service**, **source** (Logs or Traces), **first seen**, **last seen**, and total **occurrence count** +- An **Occurrences over time** chart showing how frequently the issue has occurred across the selected time range +- A **Stacktrace** section with the error type and the full call stack for pinpointing the origin of the error +- An **Infrastructure** section showing the service name and Kubernetes context — cluster, container, deployment, namespace, node, pod, and replicaset details + +### Related Errors + +The **Related Errors** section at the bottom lists individual occurrences of the issue with timestamps, service names, and direct **Show Logs** links for deeper investigation into each occurrence. + +![Infrastructure context and Related Errors table with timestamps and Show Logs links](./images/issues-3.png) + +### Actions + +From the issue detail page you can take the following actions using the action bar: + +![Action bar showing Mark as resolved, Ignore, Ask Assistant, Create in Jira, and Comments](./images/issues-4.png) + +- **Mark as resolved** — removes the issue from the Unresolved list. +- **Ignore** — mutes future notifications for this issue if it reoccurs. +- **Ask Assistant** — automatically analyses the issue and provides a concise summary, probable root cause, and recommended actions for faster resolution. +- **Create in Jira** — creates a Jira ticket directly from the issue for tracking in your existing workflow. +- **Assign** — assign the issue to one or more team members using the assign icon on the right. Assigned users are notified when an issue is assigned to them. +- **Comments** — add comments using the comment icon on the right. You can mention other users in comments, and all participating users receive notifications for any updates on the issue. + +You can also mark issues as resolved or ignored in bulk by selecting multiple issues from the main list and applying the action collectively. + +## Issue Statuses + +- **Unresolved** — an issue that has been detected and is yet to be reviewed or addressed. +- **Resolved** — an issue that has been reviewed and marked as resolved. It will no longer appear in the Unresolved list. +- **Ignored** — an issue that has been intentionally muted. No notifications will be sent if the same issue reoccurs in the future. + +## Notification Preferences + +Click **Notification settings** on the Issues dashboard to configure when you receive alerts. You can choose to be notified for: + +- **New Issues** — issues detected for the first time. +- **New and Regression Issues** — issues detected for the first time, as well as issues that had previously occurred and have reappeared. + +## User Permissions + +Access to issue actions is determined by the user's assigned role: + +- **Owner and Developer** — can mark issues as resolved or ignored, assign issues to team members, create Jira tickets, and leave comments. +- **Viewer** — can assign issues to team members and leave comments, but cannot mark issues as resolved or ignored. + +## Related Resources + +- [Notification routing](./notification-routing/) +- [Routing Rules](../alerts/routing-rules/) +- [Notification Channels](../platform/settings/notification-channels/) diff --git a/src/content/docs/docs/issues/notification-routing.mdx b/src/content/docs/docs/issues/notification-routing.mdx new file mode 100644 index 00000000..bd27668c --- /dev/null +++ b/src/content/docs/docs/issues/notification-routing.mdx @@ -0,0 +1,51 @@ +--- +title: "Notification Routing" +description: "Route error-tracking issue notifications via the dedicated Issues notification settings." +sidebar: + label: "Notification Routing" + order: 2 +--- + +Configure where issue notifications land — by service, by occurrence type (new vs. regression), and across the channels you've set up. + +## Where to find it + +Open the **Issues** list page. The header has a **Notification settings** button. + +![Issues list with Notification settings button](./images/issues-notification-settings.png) + +Clicking it opens the Issues-scoped notification settings page: + +- `<workspace>/issues/notifications` — list of all issue notification policies. +- `<workspace>/issues/notifications/create` — create a new policy. +- `<workspace>/issues/notifications/<id>` — edit a policy. + +## Anatomy of a policy + +Each policy is workspace-scoped (not per-issue) and has three fields: + +| Field | What it does | +|---|---| +| **Name** | How the policy shows up in the list. | +| **Conditions** | One or more rows. Each picks an **Issue type** (`New issues` or `New and regression`) and one or more **Target services**. | +| **Notification channels** | One or more channels that receive the notification. | + +A **Disabled** toggle lets you save the policy without activating it. + +## Create an issue notification policy + +1. Open **Issues → Notification settings** and click **New policy**. +2. Name the policy. +3. Add one or more conditions. Select the **Issue type** (`New issues` or `New and regression`) and one or more **Target services** from the autocomplete (suggestions come from services emitting telemetry). +4. Select one or more **Notification channels** from the multi-select. +5. Click **Save**. + +## Forward-looking note + +This page is a dedicated settings surface for issues because the grouping engine and Routing Rules currently handle alert signals only. Issues may converge onto the routing-rules engine in a future release; if that happens, existing issue notification policies will migrate automatically — no recreation required. + +## Related + +- [Issues](../) — the error-tracking surface this routes from. +- [Routing Rules](../../alerts/routing-rules/) — separate surface for alert notifications. +- [Notification Channels](../../platform/settings/notification-channels/) — set up where notifications land. diff --git a/src/content/docs/docs/kloudmate-assistant/images/investigations-1.jpeg b/src/content/docs/docs/kloudmate-assistant/images/investigations-1.jpeg index 44b57897..d7649cc8 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/investigations-1.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/investigations-1.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/investigations-1.png b/src/content/docs/docs/kloudmate-assistant/images/investigations-1.png index 1337e65c..44c849b2 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/investigations-1.png and b/src/content/docs/docs/kloudmate-assistant/images/investigations-1.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.jpeg b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.jpeg index 3db3b884..8db855d2 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png index e565657b..93933ed3 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png and b/src/content/docs/docs/kloudmate-assistant/images/investigations-2.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg index c8434d3d..7f79907a 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png index 65978a18..6fd768ea 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png and b/src/content/docs/docs/kloudmate-assistant/images/mcp-integrations-1.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/prompts-1.jpeg b/src/content/docs/docs/kloudmate-assistant/images/prompts-1.jpeg index 93f9d355..3c8b918e 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/prompts-1.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/prompts-1.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/prompts-1.png b/src/content/docs/docs/kloudmate-assistant/images/prompts-1.png index 71522539..7f16f0e0 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/prompts-1.png and b/src/content/docs/docs/kloudmate-assistant/images/prompts-1.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/skills-1.jpeg b/src/content/docs/docs/kloudmate-assistant/images/skills-1.jpeg index 3e0e11aa..6fb5c4f9 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/skills-1.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/skills-1.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/skills-1.png b/src/content/docs/docs/kloudmate-assistant/images/skills-1.png index 6142f070..56c2919c 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/skills-1.png and b/src/content/docs/docs/kloudmate-assistant/images/skills-1.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/skills-2.jpeg b/src/content/docs/docs/kloudmate-assistant/images/skills-2.jpeg index 7579ae77..65882ae7 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/skills-2.jpeg and b/src/content/docs/docs/kloudmate-assistant/images/skills-2.jpeg differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/skills-2.png b/src/content/docs/docs/kloudmate-assistant/images/skills-2.png index c902d921..449cde9d 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/skills-2.png and b/src/content/docs/docs/kloudmate-assistant/images/skills-2.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/images/usage-1.png b/src/content/docs/docs/kloudmate-assistant/images/usage-1.png index d458f9d1..7bc8662c 100644 Binary files a/src/content/docs/docs/kloudmate-assistant/images/usage-1.png and b/src/content/docs/docs/kloudmate-assistant/images/usage-1.png differ diff --git a/src/content/docs/docs/kloudmate-assistant/index.mdx b/src/content/docs/docs/kloudmate-assistant/index.mdx index e4a4bdc7..9b0b8e97 100644 --- a/src/content/docs/docs/kloudmate-assistant/index.mdx +++ b/src/content/docs/docs/kloudmate-assistant/index.mdx @@ -5,10 +5,18 @@ sidebar: label: "Overview" order: 1 --- -KloudMate Assistant is an AI-driven observability assistant designed to simplify how you monitor, analyze, and operate your systems. Instead of navigating multiple dashboards or writing complex queries, you can interact using natural language, whether it is "show CPU usage," "analyze recent errors," or "create a dashboard for my Kubernetes cluster." +KloudMate Assistant is an AI-driven observability assistant designed to simplify how you monitor, analyze, and operate your systems. Instead of navigating multiple dashboards or writing complex queries, you can interact using natural language — for example, "show CPU usage," "analyze recent errors," or "create a dashboard for my Kubernetes cluster." The assistant works across your entire observability stack, including metrics, logs, traces, profiles, and connected data sources. It interprets context, correlates signals, and surfaces actionable insights, reducing the need to manually switch between tools or deeply understand each underlying system. +### What KloudMate Assistant Can Do + +KloudMate Assistant supports three core modes of operation: + +- **KloudMate Assistant:** Provides real-time guidance and answers across the KloudMate platform. Ask questions, explore your data, and get instant insights through a conversational interface. +- **KloudMate Dashboarding:** Drafts dashboards and alerts on your behalf. For example, you can say "Create a dashboard for Kubernetes" and the Assistant generates a structured, ready-to-use dashboard. +- **KloudMate Investigator:** Collects an investigation brief and launches a background root cause analysis (RCA) workflow for serious incidents. The Investigator runs asynchronously and surfaces findings when complete. + ### Accessing KloudMate Assistant To access KloudMate Assistant, click the **KloudMate Assistant** icon in the top navigation bar. This opens the assistant panel where you can start a chat, run investigations, and manage settings. The assistant is also accessible from the pages listed below: diff --git a/src/content/docs/docs/kloudmate-assistant/investigations.mdx b/src/content/docs/docs/kloudmate-assistant/investigations.mdx index 17f8c855..6bf05b1c 100644 --- a/src/content/docs/docs/kloudmate-assistant/investigations.mdx +++ b/src/content/docs/docs/kloudmate-assistant/investigations.mdx @@ -4,7 +4,7 @@ description: "Use Investigations as your primary workspace to diagnose issues, e sidebar: order: 2 --- -Investigations serves as your primary workspace for diagnosing issues, exploring anomalies, and uncovering insights. You can ask questions across metrics, logs, and traces, and iteratively troubleshoot problems in a conversational flow. +Investigations is your primary workspace for diagnosing issues, exploring anomalies, and uncovering insights. You can ask questions across metrics, logs, and traces, and iteratively troubleshoot problems in a conversational flow. When you land on the Investigations page, the main area presents three quick-access cards to help you get started: @@ -22,30 +22,30 @@ Available context categories include: - **Services:** Attach APM-instrumented services you want the Assistant to focus on. - **Dashboards:** Link workspace dashboards to provide visual context. -- **Alarms:** Include alert rules relevant to your query. +- **Alerts:** Include alert rules relevant to your query. - **K8s Resources:** Reference Kubernetes Clusters, Namespaces, Deployments, and Pods. - **Synthetic Checks:** Attach uptime or browser monitors. - **Saved Queries:** Include saved log, trace, or metric queries to anchor the investigation. ## Triggering an Investigation -The **KloudMate Assistant** can quickly investigate issues or errors from logs and traces, summarize their causes, and suggest probable root causes to speed up resolution. This feature significantly reduces **root cause analysis (RCA)** time for engineering teams. +The **KloudMate Assistant** can quickly investigate issues or errors from logs and traces, summarize their causes, and suggest probable root causes to speed up resolution. This reduces **root cause analysis (RCA)** time for engineering teams. ### Decoding Issues The Issues section in KloudMate aggregates all errors and warnings detected from logs and traces into a unified dashboard view. -Navigate to any specific issue’s details page and click the “**Investigate with KloudMate Assistant**” button for instant analysis. +Navigate to any issue’s details page and click **Investigate with KloudMate Assistant** for instant analysis. The Assistant decodes the issue context, explains what went wrong, and provides actionable steps for probable RCA, including potential service impacts. ### Analyzing Error Logs and Traces The **Logs/Traces** page displays all ingested telemetry data with severity-based filtering to isolate errors quickly. -Select any error log or trace to open its details view, then use the “***Investigate with KloudMate Assistant***” option. +Select any error log or trace to open its details view, then click **Investigate with KloudMate Assistant**. The Assistant analyzes the full context, including stack traces, attributes, and timestamps, to deliver an overview of the problem, its service impact, and step-by-step fix recommendations. ### Key Benefits -- **Faster Troubleshooting:** Real-time AI insights reduces MTTR by automating initial diagnostics. +- **Faster Troubleshooting:** Real-time AI insights reduce MTTR by automating initial diagnostics. - **Unified Visibility:** Correlates logs, traces, and issues without switching contexts. - **Actionable Guidance:** Delivers plain-language root cause hypotheses and remediation steps for immediate use. @@ -67,7 +67,7 @@ This shows all past and active investigations in your workspace. Each row in the ## Starting a New Investigation -To start a new investigation, click the **New Investigation** button in the top-right corner of the Investigations page. This brings you to the Investigations chat interface where you can describe the issue or symptom you want analyzed in the input bar at the bottom of the screen. The KloudMate Investigator agent will collect your brief and begin a background RCA workflow. +To start a new investigation, click the **New Investigation** button in the top-right corner of the Investigations page. This brings you to the Investigations chat interface where you can describe the issue or symptom you want analyzed in the input bar at the bottom of the screen. The KloudMate Investigator collects your description and begins a background RCA workflow. ## Viewing an Investigation diff --git a/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx b/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx new file mode 100644 index 00000000..23f8c826 --- /dev/null +++ b/src/content/docs/docs/kloudmate-assistant/mcp-server.mdx @@ -0,0 +1,232 @@ +--- +title: "KloudMate MCP Server" +description: "Connect your AI agents and coding tools to KloudMate over MCP to query logs, traces, metrics, and alerts in plain language." +sidebar: + label: "MCP Server" + order: 5 +--- + +import { Tabs, TabItem, Steps, LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The KloudMate MCP server lets your AI coding tools and agents pull observability data straight from your workspace. Point a client like Claude Code, Cursor, or VS Code at it, and you can ask about errors, latency, traces, and firing alerts in plain language — without leaving your editor or opening the KloudMate UI. + +This is the mirror image of the [MCP integrations](../settings/#mcp-integrations) you connect *into* the Assistant. There, KloudMate is the client calling out to other tools. Here, KloudMate is the server, and your agent is the client calling in. The connection is built on the [Model Context Protocol](https://modelcontextprotocol.io/), an open standard for giving language models access to external tools and data. + +Once you're connected, your agent can chain KloudMate's query tools to investigate an incident, answer a question about a service, or run a repeatable workflow you've defined — all grounded in your real telemetry instead of guesswork. + +## Connection details + +The server is a single remote endpoint. You authenticate with a KloudMate [User Key](../../platform/settings/api-keys/#user-key) passed as a bearer token. + +| Setting | Value | +|---|---| +| **Endpoint** | `https://api.kloudmate.com/mcp` | +| **Transport** | Streamable HTTP | +| **Authentication** | `Authorization: Bearer YOUR_API_KEY` | + +The API key determines which workspace the connection reads from, so you never specify a workspace ID — the key already scopes it. To connect a different workspace, use a key that belongs to that workspace. + +## Before you start + +You'll need two things: + +- **A KloudMate User Key.** Under **Settings → API Keys**, click **Add New** and choose the **User Key** type — it's tied to your account's permissions and reads whatever you can read. Copy it when it's shown (it's shown only once) and keep it handy for your client's config. See [API Keys](../../platform/settings/api-keys/#user-key) for the full walkthrough. Creating a key requires the Admin role. +- **An MCP-capable client.** Any tool that supports remote MCP servers over Streamable HTTP with custom headers works. The steps below cover Claude Code, Cursor, VS Code, and Claude Desktop. + +:::caution +Your API key carries your access to the workspace. Treat it like a password: don't paste it into shared files or commit it to version control. The client configs below show how to keep it in an environment variable or a prompted secret instead of hardcoding it. If a key leaks, revoke it from **Settings → API Keys**. +::: + +## Connect your client + +Pick your tool and follow the steps. Swap `YOUR_API_KEY` for the key you just created. + +<Tabs> +<TabItem label="Claude Code"> + +Add the server from your terminal with the `claude mcp add` command: + +```bash +claude mcp add --transport http kloudmate https://api.kloudmate.com/mcp \ + --header "Authorization: Bearer YOUR_API_KEY" +``` + +By default the server is available in the current project only. Add `--scope user` before the server name to make it available across all your projects: + +```bash +claude mcp add --transport http --scope user kloudmate https://api.kloudmate.com/mcp \ + --header "Authorization: Bearer YOUR_API_KEY" +``` + +Run `/mcp` inside Claude Code to confirm `kloudmate` shows as connected and to see the tools it exposes. + +</TabItem> +<TabItem label="Cursor"> + +Create or edit `~/.cursor/mcp.json` (global, all projects) or `.cursor/mcp.json` (this project only), and add the server: + +```json +{ + "mcpServers": { + "kloudmate": { + "url": "https://api.kloudmate.com/mcp", + "headers": { + "Authorization": "Bearer ${env:KLOUDMATE_API_KEY}" + } + } + } +} +``` + +The `${env:KLOUDMATE_API_KEY}` reference reads the key from your environment so it never lands in the file. Set `KLOUDMATE_API_KEY` in your shell, then open **Cursor Settings → MCP** to confirm `kloudmate` connects. You can also paste the key directly in place of the reference, but keep that file out of any repository. + +</TabItem> +<TabItem label="VS Code"> + +Create `.vscode/mcp.json` in your workspace. The `inputs` block prompts you for the key the first time the server starts and stores it securely, so the key stays out of the file: + +```json +{ + "inputs": [ + { + "type": "promptString", + "id": "kloudmate-api-key", + "description": "KloudMate User API Key", + "password": true + } + ], + "servers": { + "kloudmate": { + "type": "http", + "url": "https://api.kloudmate.com/mcp", + "headers": { + "Authorization": "Bearer ${input:kloudmate-api-key}" + } + } + } +} +``` + +Start the server from the **Start** action that appears above the entry, then enter your key when prompted. The tools become available to agent mode in Copilot Chat. + +</TabItem> +<TabItem label="Claude Desktop"> + +Claude Desktop's custom-connector UI is built around OAuth and has no field for a bearer token, so connect through the config file using the [`mcp-remote`](https://github.com/geelen/mcp-remote) bridge instead. + +Open `claude_desktop_config.json` (**Settings → Developer → Edit Config**) and add: + +```json +{ + "mcpServers": { + "kloudmate": { + "command": "npx", + "args": [ + "-y", + "mcp-remote", + "https://api.kloudmate.com/mcp", + "--header", + "Authorization:${AUTH_HEADER}" + ], + "env": { + "AUTH_HEADER": "Bearer YOUR_API_KEY" + } + } + } +} +``` + +The `Bearer ` token goes in the `AUTH_HEADER` environment variable, and the `--header` argument has no spaces around the colon. That split is deliberate: it sidesteps a known bug where Claude Desktop mangles spaces inside `args` when it launches `npx`. Quit and reopen Claude Desktop fully — it reads the config only at startup — then check that `kloudmate` appears in the tools menu. + +The config file lives at: + +- **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json` +- **Windows:** `%APPDATA%\Claude\claude_desktop_config.json` + +</TabItem> +</Tabs> + +Using a different client? Point it at `https://api.kloudmate.com/mcp` with an `Authorization: Bearer YOUR_API_KEY` header. Any client that speaks Streamable HTTP and lets you set request headers can connect. + +## Verify the connection + +Once the server is connected, your client lists the KloudMate tools and your agent can call them. The quickest check is to ask a question that needs live data: + +> Which alerts are firing in KloudMate right now? + +If the connection is healthy, the agent calls a KloudMate tool and answers from your workspace. If it reports an authentication failure, the key is likely wrong, revoked, or missing the `Bearer ` prefix — recheck the header and try again. + +## Available tools + +Every tool is read-only. The server queries your telemetry; it never creates, edits, or deletes KloudMate resources. Tools accept relative time ranges like `now-1h`, `now-15m`, or `now`, as well as ISO-8601 timestamps, and default to the last hour when you don't give one. + +**Issues and alerts** + +- `search_issues` — errors and warnings aggregated from logs and traces. +- `search_alarm_rules`, `search_alarm_states`, `search_alarm_states_history` — list alert rules, see what's firing now, and review how states changed over time. + +**Service health (APM)** + +- `get_service_error_count`, `get_service_requests_count_and_avg_latency` — error counts, throughput, and average latency for a service. +- `get_service_to_service_requests_count`, `get_service_to_service_failed_requests_count`, `get_service_to_service_request_duration` — request volume, failures, and latency between services, to scope a blast radius. + +**Logs, traces, and spans** + +- `search_logs`, `search_traces`, `search_spans` — query telemetry with filters. +- `get_log_details`, `get_span_details`, `get_trace_spans`, `get_trace_logs` — pull the full record for a single log line, span, or trace. + +**Schema discovery** + +- `discover_dataset_fields`, `discover_field_values` — find which fields and values exist before building a filter. +- `discover_metrics`, `discover_metric_label_values` — find metric names and the values their labels take. + +**Aggregations and queries** + +- `run_aggregation_query`, `run_metric_aggregation_query` — grouped counts, percentiles, and timeseries over logs and traces, or over metrics. +- `run_multiple_log_trace_aggregations`, `run_multiple_metric_aggregations` — run several aggregations in one call. + +**Dashboards** + +- `search_dashboards`, `get_dashboard_panels`, `run_dashboard_panel_query` — find a dashboard, inspect its panels, and run a panel's query. + +**Kubernetes** + +- `list_kubernetes_resources`, `search_kubernetes_events` — inspect clusters, namespaces, deployments, and pods, and read recent cluster events. + +**Synthetics** + +- `search_synthetic_monitors`, `search_synthetic_incidents` — check uptime and browser monitors and the incidents they've raised. + +## Example prompts + +Once connected, ask in plain language. The agent picks the right tools and fills in the queries: + +- *"What are the top errors in the `checkout` service over the last hour?"* +- *"Show me the slowest traces for `api-gateway` in the past 15 minutes."* +- *"Which alerts are firing right now, and what changed in the half hour before they fired?"* +- *"Pull the logs for trace `4bf92f3577b34da6` and summarize where it failed."* +- *"Compare request volume between the `frontend` and `cart` services this morning."* + +## Build custom workflows + +Because your agent can chain these tools and combine them with whatever else it can reach — your codebase, a runbook, your deployment history — you can codify investigations that used to mean clicking through several screens. + +A triage workflow, for example, might pull the currently firing alerts, scope the affected services with the service-to-service tools, gather the relevant traces and logs, and hand you a single summary with a probable cause. Save it as a reusable command or prompt in your tool of choice, and the next on-call run is one sentence instead of twenty clicks. + +## Security and scope + +A few things worth keeping in mind: + +- **Read-only.** The server can't change anything in your workspace. Connecting it can't cause an outage or alter a dashboard. +- **Workspace-scoped.** The key binds the connection to one workspace. Use separate keys for separate workspaces. +- **Keep the key secret.** Prefer the environment-variable or prompted-secret options above over pasting the key into a file. Add config files that contain a key to `.gitignore`. +- **Revoke when needed.** If a key is exposed or no longer used, revoke it from **Settings → API Keys**. Any client using it stops working immediately. + +## Related + +<CardGrid> + <LinkCard title="API Keys" description="Create the User Key the MCP server authenticates with, and manage ingest keys." href="../../platform/settings/api-keys/#user-key" /> + <LinkCard title="Assistant Settings" description="Connect external MCP servers and tools into the KloudMate Assistant." href="../settings/#mcp-integrations" /> + <LinkCard title="Investigations" description="Run AI-assisted root cause analysis inside KloudMate." href="../investigations/" /> + <LinkCard title="KloudMate Assistant" description="The in-app AI assistant for monitoring, dashboards, and RCA." href="../" /> +</CardGrid> diff --git a/src/content/docs/docs/llm-observability/ai-trace-observability.mdx b/src/content/docs/docs/llm-observability/ai-trace-observability.mdx new file mode 100644 index 00000000..dc63fb34 --- /dev/null +++ b/src/content/docs/docs/llm-observability/ai-trace-observability.mdx @@ -0,0 +1,64 @@ +--- +title: "AI Trace Observability" +description: "Read prompts, tool calls, token usage, and finish reasons on an AI trace in KloudMate — the conversation view, AI summary, and AI Flow graph." +sidebar: + order: 2 +--- + +When a trace contains GenAI spans, KloudMate adds AI-specific views on top of the normal trace detail. You get a summary of the run, the full conversation behind each model call, token and tool details, and a graph of the agent's steps — without digging through raw span attributes. + +To get there, open **Traces**, find a trace from your LLM service, and open it. The sections below describe what you'll see. + +## AI summary + +At the top of an AI trace, the **AI summary** gives you the shape of the run at a glance: the models used, how many model and tool calls ran, total tokens (with cached input called out), the time split between model and tool calls, and how many AI spans errored. + +![AI summary, AI Flow, and waterfall for an agent trace](./images/llm-overview.png) + +Use it to answer the first questions you usually have about an agent run — which model ran, how many tokens it spent, and whether the time went into the model or the tools. + +## Conversation view + +Select a model span and KloudMate opens the **Conversation** tab: the prompt and response rendered as a transcript instead of a wall of JSON. Each turn is labelled by role — system, user, assistant, or tool — and shown in order. + +![The Conversation tab showing an assistant turn calling a tool](./images/llm-conversation.png) + +The transcript captures the parts that matter when an answer looks wrong: + +- **Assistant tool calls** appear inline with the tool name and the arguments the model passed. +- **Tool results** show what each tool returned. +- **The finish reason** for the turn is shown next to the role, so a truncated or filtered response stands out. +- **Copy** any turn with the button in its header. + +This is the fastest way to see what the model was actually told and what it actually said. + +## Token, cache, and finish-reason details + +The **Gen AI** tab holds the structured details for a model or tool span: + +- **Tokens** as a single line — total, input (with cached input called out), and output. Cached tokens are part of the input count, not added on top, so the number reflects what you were billed for. +- **Finish reason** chips. A normal `stop` reads as neutral; an abnormal reason like `length` (truncated) or `content_filter` is highlighted so you don't miss it. +- **Request parameters** the call used, such as temperature, top-p, and max tokens. +- **Tool definitions** offered to the model on that call, collapsed by default. + +For tool spans, the Gen AI tab shows the **tool call** — its arguments and result — as structured JSON. The full set of raw `gen_ai.*` attributes stays available under the **Attributes** tab. + +## AI Flow + +Switch the view mode to **AI Flow** (top right of the trace) to read the run as a workflow graph rather than a time-ordered waterfall. AI Flow shows only the GenAI spans, laid out by how the agent's steps connect. + +Each node shows the step's kind (LLM, tool, agent, and so on), the model, token usage including cached tokens, and a badge when a call finished abnormally or errored. Click a node to open its span detail — the same Conversation and Gen AI tabs described above. For long, linear agent chains, this is easier to scan than the waterfall. + +## AI cues in the waterfall + +The standard timeline stays AI-aware. GenAI spans get distinct icons by kind — model calls, tool calls, agents, and retrieval steps each read differently — and model spans show the model name inline on the row. That makes the LLM and tool steps easy to pick out among the rest of a trace's spans. + +## Where the data comes from + +These views are built from standard OpenTelemetry GenAI attributes, so they light up for any compatible instrumentation. The conversation view in particular depends on the instrumentation capturing prompt and response content. To produce traces like the ones above, see [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/). + +## Related resources + +- [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/) +- [Introduction to OpenLLMetry](../what-is-openllmetry/) +- [Trace Detail](/apm-and-tracing/trace-detail/) — the trace views that AI observability builds on diff --git a/src/content/docs/docs/llm-observability/images/llm-conversation.png b/src/content/docs/docs/llm-observability/images/llm-conversation.png new file mode 100644 index 00000000..cb737d2d Binary files /dev/null and b/src/content/docs/docs/llm-observability/images/llm-conversation.png differ diff --git a/src/content/docs/docs/llm-observability/images/llm-overview.png b/src/content/docs/docs/llm-observability/images/llm-overview.png new file mode 100644 index 00000000..c5faec82 Binary files /dev/null and b/src/content/docs/docs/llm-observability/images/llm-overview.png differ diff --git a/src/content/docs/docs/llm-observability/index.mdx b/src/content/docs/docs/llm-observability/index.mdx index 52267e2f..0acb19f6 100644 --- a/src/content/docs/docs/llm-observability/index.mdx +++ b/src/content/docs/docs/llm-observability/index.mdx @@ -1,18 +1,33 @@ --- title: "LLM Observability" -description: "Monitor, trace, and optimize large language models (LLMs) and generative AI applications using KloudMate." +description: "Trace, debug, and understand LLM and agent applications in KloudMate — prompts, tool calls, token usage, and latency." sidebar: label: "Overview" order: 1 --- -LLM-based applications leverage the deep learning capabilities of GenAI and Large Language Models (LLMs) to understand, interpret, and generate human-like text and reasoning. +A single answer from an LLM agent can hide several model calls, a handful of tool calls, and a retry or two. When that answer is wrong, slow, or expensive, a flat log line won't tell you why — you need to see the whole chain. -Due to the massive scale of data processing involved—coupled with distributed architectures and real-time interactions—monitoring LLM applications requires sophisticated systems capable of accessing high volumes of data, performing end-to-end tracing, and providing insights into costs, token usage, and latency. +KloudMate traces LLM and agent applications as part of its OpenTelemetry-based tracing. Each run becomes one distributed trace, so you can read the model's reasoning, every tool call, the tokens each call spent, and where the time went — alongside the rest of your application's traces, in the same place. -KloudMate provides native support for LLM Observability via the OpenTelemetry ecosystem. +## What you can do -## In This Section +- **Read the full conversation** behind a span — system, user, assistant, and tool turns — without reconstructing it from raw attributes. +- **See token usage** per call and per trace, including cached input tokens, so you can find the calls driving cost. +- **Follow the agent's tools** — which tools it called, with what arguments, and what they returned. +- **Catch failures and truncations** — tool errors and abnormal finish reasons (`length`, `content_filter`) are surfaced, not buried. +- **Read the run as a graph** with AI Flow, which lays out the model and tool steps as a workflow. +- **Attribute traces to a user or session** when your instrumentation tags them, so you can follow one customer or one conversation. -- [Introduction to OpenLLMetry](./what-is-openllmetry/) -- [Instrument Python Apps with OpenLLMetry](./instrument-with-openllmetry/) +[AI Trace Observability](./ai-trace-observability/) covers each of these in detail. + +## Getting data in + +KloudMate reads standard OpenTelemetry GenAI spans, so any OpenTelemetry-compatible instrumentation works. The fastest path is [OpenLLMetry](./what-is-openllmetry/), which auto-instruments common frameworks and providers. + +To instrument an app from scratch, follow [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/). + +## In this section + +- [AI Trace Observability](./ai-trace-observability/) — what KloudMate shows you on an AI trace +- [Introduction to OpenLLMetry](./what-is-openllmetry/) — how the data is captured diff --git a/src/content/docs/docs/llm-observability/instrument-with-openllmetry.mdx b/src/content/docs/docs/llm-observability/instrument-with-openllmetry.mdx deleted file mode 100644 index 34587423..00000000 --- a/src/content/docs/docs/llm-observability/instrument-with-openllmetry.mdx +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: "Instrument Python Apps with OpenLLMetry" -description: "Learn how to instrument a Python application using OpenLLMetry and send traces to KloudMate." -sidebar: - order: 3 ---- -In this guide, we will walk you through the process of setting up and using OpenLLMetry in Python. You can use the following steps to instrument a Python program with OpenLLMetry and then emit and visualize its traces within [KloudMate](https://app.kloudmate.com). - -The instrumentation demonstrated in this guide lets OpenLLMetry capture and send OpenAI model KPIs to KloudMate, where there is a dedicated '**Traces** ' section to visualize the captured data. In addition to this OpenLLMetry also has instrumentations for vector databases such as **Pinecone** , and LLM frameworks such as **LangChain**. Feel free to adapt the instructions to your preferred framework. - -## 1. Prerequisites - -- Ensure that you have Python 3 installed on your local machine -- Create an OpenAI account and get your OpenAI API key ([here's how](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key)) -- Create a [KloudMate account](https://app.kloudmate.com/signup) and generate your KloudMate API Key - -:::info -The KloudMate API Key can be obtained by logging into your KloudMate account and navigating to **Settings >> API Keys**. -::: - -## 2. Example Application -For this setup, we will be using a basic Python program that uses OpenAI API. Adhere to the steps outlined below to set up the environment for the program. - -## 3. Installation - -1. Create a new directory and activate a virtual environment: - -```bash -mkdir openLLMetry-getting-started -cd openLLMetry-getting-started -python3 -m venv venv -source ./venv/bin/activate -``` - -2. Next, install the `traceloop-sdk` and `openai`. - -```bash -pip install traceloop-sdk openai -``` - -## 4. Instrumentation - -1. Create a file named `app.py` and add the following code to it: - -```python -from traceloop.sdk import Traceloop -from openai import OpenAI -from traceloop.sdk.decorators import workflow, task -import time - -client = OpenAI(api_key="<your-openai-api-key>") - -Traceloop.init( - app_name="your app name", - api_endpoint="https://otel.kloudmate.com:4318", - headers={"Authorization": "<your-kloudmate-api-key>"} -) - -@task(name="joke_creation") -def create_joke(): - completion = client.chat.completions.create( - model="gpt-3.5-turbo", - messages=[{"role": "user", "content": "Tell me a joke about OpenTelemetry"}], - ) - - return completion.choices[0].message.content - -@workflow(name="pirate_joke_generator") -def joke_workflow(): - eng_joke = create_joke() - print(eng_joke + "\n\n") - -joke_workflow() - -time.sleep(10) -``` - -:::info -Ensure you replace `<your-openai-api-key>` and `<your-kloudmate-api-key>` with your actual API keys. -::: - -## 5. Run the Instrumented Program - -1. Use the following command to run the program - -```bash -python3 app.py -``` - -## 6. Visualize Traces - -1. Login to [KloudMate](https://app.kloudmate.com) and navigate to the '**Traces** ' section. - -- You will be able to view the LLM attributes as shown below: - -![image](./images/llm-observability-with-kloudmate-and-openllmetry-1.png) - -- You will also be able to visualize the traces and spans: - -![image](./images/llm-observability-with-kloudmate-and-openllmetry-2.png) - -*** - -## Related Resources - -- [Introduction to OpenLLMetry](../what-is-openllmetry/) -- [What Is OpenTelemetry?](../../opentelemetry/what-is-opentelemetry/) - - diff --git a/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx b/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx index bf5ec4b2..05e94034 100644 --- a/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx +++ b/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx @@ -1,39 +1,36 @@ --- title: "Introduction to OpenLLMetry" -description: "Understand how OpenLLMetry extends OpenTelemetry for AI and LLM observability." +description: "How OpenLLMetry extends OpenTelemetry to capture traces from LLM and agent applications." sidebar: - order: 2 + order: 3 --- -LLM-based applications leverage the deep learning capabilities of GenAI and LLMs (Large language models) to understand, interpret, and generate human-like text and reasoning, enabling tasks such as language translation, text summarization, sentiment analysis, question answering, chatbot interactions, content generation, and more. -Since LLMs typically involve intricate architectures with numerous layers of neural networks, gaining insight into their internal processes and performance metrics is highly challenging. Additionally, because of the massive scale of data processing involved in LLM applications (coupled with their distributed nature and real-time interactions), the monitoring of LLM applications requires sophisticated monitoring systems capable of accessing high volumes of data, performing high-end tracing, and providing timely insights. +LLM applications are built from model calls, tool calls, and framework glue, and their behavior shifts with every prompt. To debug and tune them, you need the same tracing you'd use for any distributed system — adapted to capture model-specific details like prompts, token usage, and tool calls. -The inherent complexity, scale, and diversity of LLMs, mandate the use of solutions that can provide a unified, tailored, and vendor-neutral approach to observability. And at the forefront, is the CNCF-based project, the [OpenTelemetry](https://opentelemetry.io/) framework. +[OpenTelemetry](https://opentelemetry.io/) is the open standard for that tracing. OpenLLMetry extends it to the LLM layer. +## What is OpenLLMetry? -## What Is OpenLLMetry? +[OpenLLMetry](https://www.traceloop.com/docs/openllmetry/introduction) is an open-source SDK from Traceloop that adds LLM instrumentation to OpenTelemetry. It's Apache-2.0 licensed and built on the same protocols OpenTelemetry uses, so the traces it produces flow to any OpenTelemetry backend, including KloudMate. +Because it builds on OpenTelemetry, you don't run a separate pipeline for AI telemetry. Model and tool spans sit in the same traces as your HTTP, database, and queue spans. -[OpenLLMetry](https://www.traceloop.com/docs/openllmetry/introduction) is an open-source framework developed by Traceloop, that simplifies the process of monitoring and debugging Large Language Models. It is built on top of OpenTelemetry, ensuring non-intrusive tracing and seamless integration with leading observability platforms and backends like **KloudMate.** +## How it works -**OpenLLMetry** aims to standardize the collection of mission-critical LLM metrics, spans, traces, and logs through OpenTelmetry. +OpenLLMetry ships instrumentations for the common pieces of an LLM stack: -## How Does OpenLLMetry Work? +- **Providers** such as OpenAI, Anthropic, and Cohere. +- **Frameworks** such as LangChain and LlamaIndex. +- **Vector databases** such as Pinecone and Chroma. -OpenLLMetry is an extension of OpenTelemetry, for AI-related instrumentations. It is written and developed in Python, along with Typescript, and is licensed with Apache 2.0. Since it’s based on OpenTelemetry, it uses the same protocols for instrumentation, data collection, and integration. +When you initialize the SDK, it patches these libraries so each model call, tool call, and chain step emits a span — with attributes for the prompt, response, model, and token usage. Your application logic stays the same; you add the SDK and point it at your backend. -Traceloop, the developer, has created multiple instrumentations for LLMs such as OpenAI, Anthropic, and Cohere; vector databases like Pinecone; and LLM frameworks like LangChain and Haystack. These instrumentations help in tracing LLM responses & prompts, and monitoring model performance, token usage, and other essential metrics. +![How OpenLLMetry captures and exports LLM telemetry](./images/openllmetry_-opentelemetry-based-observability-for-llms-1.png) -![image](./images/openllmetry_-opentelemetry-based-observability-for-llms-1.png) +The SDK exports over OTLP, the OpenTelemetry wire protocol, so sending data to KloudMate comes down to setting the endpoint and your API key. -Through the instrumentations for Gen-AI models and LLMs, OpenLLMetry captures and sends AI model KPIs and telemetry data to a target system (an observability backend like [KloudMate](https://www.kloudmate.com)). +## Related resources - The developer, Traceloop, also provides a ready SDK that makes it easy to use OpenLLMetry instrumentations for users who may not be familiar with OpenTelemetry. - -*** - - -## Related Resources - -- [Instrument Python Apps with OpenLLMetry](../instrument-with-openllmetry/) +- [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/) +- [AI Trace Observability](../ai-trace-observability/) - [What Is OpenTelemetry?](../../opentelemetry/what-is-opentelemetry/) diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-1.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-1.jpeg index bc5908b9..8a892d1f 100644 Binary files a/src/content/docs/docs/logs/log-explorer/images/log-explorer-1.jpeg and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-1.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-2.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-2.jpeg new file mode 100644 index 00000000..205ecc0e Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-2.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-3.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-3.jpeg new file mode 100644 index 00000000..77c24831 Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-3.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-4.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-4.jpeg new file mode 100644 index 00000000..e0653598 Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-4.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-5.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-5.jpeg new file mode 100644 index 00000000..fe89d1f9 Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-5.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-6.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-6.jpeg new file mode 100644 index 00000000..0fc77307 Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-6.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/images/log-explorer-7.jpeg b/src/content/docs/docs/logs/log-explorer/images/log-explorer-7.jpeg new file mode 100644 index 00000000..0d5410e1 Binary files /dev/null and b/src/content/docs/docs/logs/log-explorer/images/log-explorer-7.jpeg differ diff --git a/src/content/docs/docs/logs/log-explorer/index.mdx b/src/content/docs/docs/logs/log-explorer/index.mdx index 4320f674..f2ae670d 100644 --- a/src/content/docs/docs/logs/log-explorer/index.mdx +++ b/src/content/docs/docs/logs/log-explorer/index.mdx @@ -25,7 +25,7 @@ Select any log entry to inspect its full payload, labels, and metadata. Use the search bar at the top of Log Explorer to find matching log entries across your environment. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/tXoqALt4fLIyw54nOZ6nG-20251229-135256.jpeg?format=webp) +![image](./images/log-explorer-2.jpeg) Refine results further by adjusting the selected time range from the time scale menu. @@ -33,37 +33,35 @@ Refine results further by adjusting the selected time range from the time scale Use the sidebar filters to narrow the result set by relevant dimensions. Multiple filters can be combined to isolate a very specific set of logs. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/awFyZxlhDVrrw4xdhozxN-20251229-135332.jpeg?format=webp) +![image](./images/log-explorer-3.jpeg) ## Saved Queries To save a query, click the dropdown arrow next to **Saved Queries** in the top-right area of Log Explorer. Enter a query name, description, and folder, then click **Save**. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/mqglxV-yaA07fHhXsHYyk-20251229-135409.jpeg?format=webp) +![image](./images/log-explorer-4.jpeg) Saved queries remain available from the **Saved Queries** section for quick reuse. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/Rey4etpGf3pNAVB9TV23B-20251229-135657.jpeg?format=webp) - Use folders to keep saved queries organized by service, team, or use case. ## Inspect Log Entries Select any log entry to view detailed information and analysis. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/h7S8g9et6rolT0mbsN5lL-20251229-141757.png?format=webp) +![image](./images/log-explorer-5.jpeg) -For error logs, use **Investigate with KloudMate Assistant** to generate a summary, identify the probable root cause, and suggest a next step. +**Ask Assistant** opens KloudMate Assistant directly from the issue. It reviews the error log and its associated trace, generates a Log & Trace Summary, identifies the probable root cause, and suggests next steps for faster resolution. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/6Q8kOseiFXN7u1iyuEz3r-20251229-142111.png?format=webp) +![image](./images/log-explorer-6.jpeg) -Refer to the KloudMate Assistant documentation for more details. +Refer to the [KloudMate Assistant](../../kloudmate-assistant/) documentation for more details. ## Stream Logs Use **Stream** to watch logs as they are ingested into KloudMate. -![image](https://images.archbee.com/dfeZmyJsGjUXyEH2hgHVe/w5OtRcJGu8q6b-uaQKGhU-20251229-144423.jpeg?format=webp) +![image](./images/log-explorer-7.jpeg) ## Related Resources diff --git a/src/content/docs/docs/platform/homepage.mdx b/src/content/docs/docs/platform/homepage.mdx index 2096e5eb..8a4c1b18 100644 --- a/src/content/docs/docs/platform/homepage.mdx +++ b/src/content/docs/docs/platform/homepage.mdx @@ -5,14 +5,14 @@ sidebar: label: "Homepage" order: 2 --- -The KloudMate homepage is the first screen you see after signing in. It gives you a consolidated view of service health, recent issues, alarms, and usage trends so you can understand the current state of your environment at a glance. +The KloudMate homepage is the first screen you see after signing in. It gives you a consolidated view of service health, recent issues, alerts, and usage trends so you can understand the current state of your environment at a glance. ## What the Homepage Shows The homepage brings together high-level platform signals such as: - Service health and activity -- Recent issues and alarms +- Recent issues and alerts - Cost and usage insights - Key observability widgets selected by your team @@ -50,4 +50,5 @@ Use the time range selector in the top-right corner of the homepage to update th - [Settings](../settings/) - [Dashboards](../../visualize-data/dashboards/) -- [Alarms & Notifications](../../alarms/) +- [Alerts & Notifications](../../alerts/) +- [Reliability & SLOs](../../reliability/) diff --git a/src/content/docs/docs/platform/settings/api-keys.mdx b/src/content/docs/docs/platform/settings/api-keys.mdx index ff37c610..45242c9a 100644 --- a/src/content/docs/docs/platform/settings/api-keys.mdx +++ b/src/content/docs/docs/platform/settings/api-keys.mdx @@ -1,33 +1,38 @@ --- title: "API Keys" -description: "Manage API keys for data ingestion and integration with KloudMate." +description: "Create and manage API keys: User Keys for programmatic and MCP access, and Ingest Keys for sending telemetry to KloudMate." sidebar: label: "API Keys" order: 2 --- -API keys are required to authenticate and send telemetry data (logs, metrics, and traces) from your applications and infrastructure to KloudMate. +import { Steps } from '@astrojs/starlight/components'; -KloudMate uses a dedicated **API Keys** management section to give you granular control over the keys used across your organization. +API keys authenticate requests to KloudMate. You'll use them for two jobs: sending telemetry *into* your workspace, and reading data back *out* through the KloudMate API or the [MCP server](../../../kloudmate-assistant/mcp-server/). What a key can do comes down to its **type**, which you choose when you create it. -## Manage API Keys +Find your keys under **Settings → API Keys**. The page lists every key in the workspace along with its type, note, and creation date. A key belongs to the workspace it's created in, so create it in the workspace whose data you want to reach. Adding a key requires the **Admin** role. -You can create, view, and revoke API keys by navigating to **Settings -> API Keys** from the main sidebar. +## Key types -### Create an API Key +KloudMate has three key types. Pick the one that matches what you're building. -1. Navigate to **Settings -> API Keys**. -2. Click the **Add Key** or **Create API Key** button. -3. Provide a recognizable name for the key to identify its purpose or the environment it will be used in (e.g., `prod-kubernetes-cluster` or `frontend-app`). -4. Click **Create** or **Save**. +### User Key -Once created, the new API Key will be displayed. **Copy this key immediately**, as you will need it to configure your OpenTelemetry collectors, SDKs, or the KloudMate agent. +A User Key works for any KloudMate API operation and is tied to **your** account's permissions — it can read whatever you can read. This is the key to use for the [KloudMate MCP server](../../../kloudmate-assistant/mcp-server/), and for any script or service that queries your observability data through the API. -### Using API Keys +Pass it as a bearer token: -When configuring your instrumentation, you will typically need to provide your API key in the headers of your telemetry exports. +```http +Authorization: Bearer YOUR_API_KEY +``` + +Because the key carries your access, treat it like a password. Anyone holding it can act as you in the workspace. + +### Ingest Key — Backend -For OpenTelemetry configurations (OTLP), this is generally provided in the `Authorization` header: +A Backend ingest key is for server-side telemetry. Your OpenTelemetry collectors, SDKs, and the KloudMate agent use it to ship logs, metrics, and traces. It bypasses CORS and puts no restriction on where data comes from, so keep it on your servers — never ship it in a browser or mobile app. + +For OpenTelemetry (OTLP) exporters, pass it in the `Authorization` header: ```yaml exporters: @@ -37,13 +42,37 @@ exporters: Authorization: "Bearer YOUR_API_KEY" ``` -### Revoking an API Key +### Ingest Key — Frontend + +A Frontend ingest key is for telemetry sent straight from a browser or mobile app, such as [Real-User Monitoring](../../../rum/). Because the key ships in client code where anyone can read it, you must list the **allowed hosts** it can send from. Data from any other origin is rejected, which limits the damage if the key leaks. + +## Create an API key + +<Steps> + +1. Go to **Settings → API Keys** and click **Add New**. (The button appears only if you have the Admin role.) + +2. Choose a **Key Type** — User Key, Ingest Key – Backend, or Ingest Key – Frontend. You can't change the type after the key is created, so pick the one that fits. + +3. For a Frontend ingest key, add at least one **allowed host**. Include the protocol and port if there is one, for example `https://app.example.com`. Data sent from any other origin is rejected. + +4. Add a **Note** that says what the key is for — `mcp-server`, `prod-otel-collector`, `rum-web`. The list shows only the note, so make it recognizable. + +5. Click **Submit**. + +</Steps> + +The key is shown **once**, right after you create it: + +> Please copy and save the API key below as it will not be shown again. + +Copy it then and store it somewhere safe — a secret manager, your client's config, or a CI secret. There's no way to retrieve it later; if you lose it, create a new one and delete the old. + +## Edit or delete a key -If an API key is compromised or no longer needed, you should revoke it to prevent unauthorized data ingestion. +You can edit or delete only the keys **you** created. Open the actions menu on a key's row to find both. -1. Navigate to **Settings -> API Keys**. -2. Locate the key you want to remove. -3. Click the delete from the actions menu on the row. -4. Confirm the deletion. +- **Edit** lets you change the note and, for a Frontend key, its allowed hosts. The key type is fixed once created. +- **Delete** is permanent — *this action cannot be undone*. Any agent, collector, app, or MCP client using that key stops working the moment you delete it, so swap in a replacement first if the key is in active use. -*Note: Revoking an API key will immediately stop any active agents or applications using that key from sending data to KloudMate.* +Delete a key promptly if it's exposed or no longer needed. That's the way to cut off access for a key that's been leaked. diff --git a/src/content/docs/docs/platform/settings/images/channel-km-incidents-form.png b/src/content/docs/docs/platform/settings/images/channel-km-incidents-form.png new file mode 100644 index 00000000..cf51f6df Binary files /dev/null and b/src/content/docs/docs/platform/settings/images/channel-km-incidents-form.png differ diff --git a/src/content/docs/docs/platform/settings/index.mdx b/src/content/docs/docs/platform/settings/index.mdx index 742f0661..09bececb 100644 --- a/src/content/docs/docs/platform/settings/index.mdx +++ b/src/content/docs/docs/platform/settings/index.mdx @@ -12,6 +12,7 @@ The Settings area is where you manage workspaces, users, security, billing, and - [Workspaces](./workspaces/) - [API Keys](./api-keys/) - [Users & Permissions](./users-permissions/) +- [Single Sign-On (SSO)](./sso/) - [Profile & Security](./profile-security/) - [Data Sources](./data-sources/) - [Remove an AWS Account](./deleting-your-aws-account-from-kloudmate/) diff --git a/src/content/docs/docs/platform/settings/notification-channels.mdx b/src/content/docs/docs/platform/settings/notification-channels.mdx index 3f154a81..60346686 100644 --- a/src/content/docs/docs/platform/settings/notification-channels.mdx +++ b/src/content/docs/docs/platform/settings/notification-channels.mdx @@ -7,7 +7,7 @@ sidebar: import { Tabs, TabItem } from '@astrojs/starlight/components'; -To receive notifications from KloudMate about incidents in your infrastructure, you need to set up notification channels. Notification channels are the communication platforms where alerts are sent. +To receive notifications from KloudMate, you need to set up notification channels — the communication platforms where alerts and incident updates are sent. KloudMate supports the following notification channels: - Email @@ -16,6 +16,7 @@ KloudMate supports the following notification channels: - MS Teams - Webhook - Jira +- KloudMate Incidents — bridges alerts and synthetic checks into [Incident Management](../../../incident-management/) :::info In the case of multiple workspaces and AWS accounts, you must set up notifications for each account in their respective **Settings**. @@ -46,10 +47,13 @@ Below are the configuration steps for each specific channel type: ### Slack +KloudMate uses a single unified Slack app for both notifications and IM ChatOps. Installing it once gives you threaded notifications, message attribution, and the ability to acknowledge or resolve incidents directly from Slack. + 1. Select **Slack** from the dropdown menu, then click the **Add to Slack** button. -2. You will be redirected to the Slack integrations page. Follow the on-screen instructions, add a message for your App Manager, then click **Submit**. +2. You will be redirected to Slack's OAuth flow. Pick the channel KloudMate should post into and approve the bot scopes. +3. On approval, KloudMate creates the channel row and the Slack integration is also visible under **Incidents → Settings → ChatOps**. -![image](./images/slack-1.png) +![Slack channel form](./images/slack-1.png) ### MS Teams @@ -77,7 +81,7 @@ Optionally, you can download the KloudMate logo from the link provided on the sc 1. Select **Webhook** from the dropdown menu. 2. Enter a **Webhook Name** and the **Webhook URL** in their respective fields. -3. Under **Secret**, provide a secure key. This secret will be used to generate an HMAC-SHA256 signature for each webhook payload, allowing you to verify that the request authentically came from KloudMate. +3. Under **Secret**, provide a secure key. This secret is used to generate an HMAC-SHA256 signature for each webhook payload, allowing you to verify that the request came from KloudMate. 4. Click **Submit**. ![image](./images/webhooks-1.png) @@ -146,6 +150,187 @@ Here are examples of how to perform this validation in common programming langua </TabItem> </Tabs> +#### Notification Payloads + +Every webhook is delivered as a single `POST` with a JSON body and `Content-Type: application/json`. The body you receive is exactly the bytes that were HMAC-signed (see above), so verify the signature against the raw body before parsing. + +The payload shape depends on which KloudMate feature triggered the notification: + +- **Alert group** notifications carry a top-level `event` field (`opened`, `appended`, `resolved`, or `rca_completed`) that identifies the lifecycle stage. +- The other payloads have no `event` field — identify them by their distinctive top-level keys: `slo_name` (SLO burn rate), `monitor_id` (synthetic monitor), `serviceName` (new issue), and `investigationId` (AI investigation). + +The tabs below show one representative example of each payload, with notes on how the variants differ. + +<Tabs> + <TabItem label="Alert group"> + Sent when an [alert group](../../../alerts/) opens, gains new alerts, resolves, or completes a root-cause analysis. Each rule reports its full `instances` array — no sampling, every instance in the notification is included, each tagged with its `state` — alongside per-state `counts`. Each rule also exposes `commonLabels` and `commonAnnotations` — the label and annotation values identical across all of that rule's instances — so receivers and the chat/email templates can show them once instead of on every row; the individual `instances[]` still carry their own complete `labels` and `annotations`. Both `totals` and each rule's `counts` are keyed by the user-facing state (`Firing`, `Resolved`, `No Data`, `Error`, `Normal`). `group.mode` is `"group"` for a correlated alert group and `"standalone"` for a single-alert notification (one per alarm rule). Chat and email channels cap how many instance rows they render; the webhook always carries the complete list. + + ```json + { + "event": "opened", + "emitted_at": "2026-05-23T10:00:05.123Z", + "group": { + "id": "550e8400-e29b-41d4-a716-446655440000", + "title": "High CPU on payments-api", + "state": "Open", + "severity": "sev1", + "severity_escalated": false, + "opened_at": "2026-05-23T10:00:00.000Z", + "resolved_at": null, + "labels": { "service": "payments-api", "env": "production" }, + "annotations": { "runbook": "https://wiki.example.com/runbooks/cpu" }, + "routing_rule": { "id": "rr-1", "name": "Production critical" }, + "workspace_id": "ws-1", + "signal_count": 2, + "url": "https://app.kloudmate.com/ws-1/alerts/groups/550e8400-e29b-41d4-a716-446655440000", + "mode": "group" + }, + "totals": { "Firing": 2 }, + "rules": [ + { + "alarm_id": "alarm-cpu-1", + "alarm_name": "CPU > 90%", + "commonLabels": { "service": "payments-api", "env": "production" }, + "commonAnnotations": { "runbook": "https://wiki.example.com/runbooks/cpu" }, + "counts": { "Firing": 2 }, + "instances": [ + { + "source_event_id": "3f2504e0-4f89-41d3-9a0c-0305e82c3301", + "state": "Firing", + "labels": { "service": "payments-api", "env": "production", "host": "ip-10-0-1-23" }, + "annotations": { + "runbook": "https://wiki.example.com/runbooks/cpu", + "summary": "CPU at 96% on ip-10-0-1-23" + }, + "received_at": "2026-05-23T10:00:00.000Z" + }, + { + "source_event_id": "9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d", + "state": "Firing", + "labels": { "service": "payments-api", "env": "production", "host": "ip-10-0-1-24" }, + "annotations": { + "runbook": "https://wiki.example.com/runbooks/cpu", + "summary": "CPU at 93% on ip-10-0-1-24" + }, + "received_at": "2026-05-23T10:00:02.000Z" + } + ], + "dashboardUrl": "https://app.kloudmate.com/ws-1/dashboards/db-1?panel=4" + } + ], + "rca": null + } + ``` + + **Variants by `event`:** + - `appended` — new or still-firing alerts on an already-open group; each instance carries its current `state`, and `counts`/`totals` tally them by state. + - `resolved` — `group.state` becomes `"Resolved"`, `group.resolved_at` is set, and every instance reports `"state": "Resolved"`, so `counts`/`totals` move under `Resolved`. + - `rca_completed` — the `rca` object is populated instead of `null`: + + ```json + "rca": { + "investigation_id": "inv-9", + "summary": "Deployment v1.42 introduced a regression in the payments service.", + "url": "https://app.kloudmate.com/ws-1/assistant/investigations/inv-9" + } + ``` + </TabItem> + <TabItem label="New issue"> + Sent when a new [issue](../../../issues/) is created for a service. + + ```json + { + "subject": "[FATAL] New Issue - api (workspace-name / org-name)", + "serviceName": "api", + "severity": "FATAL", + "errorMessage": "TypeError: Cannot read property 'x' of undefined", + "occurrence": 7, + "lastOccurrenceAt": "2026-05-23T09:55:00.000Z", + "actionUrl": "https://app.kloudmate.com/ws-1/issues/iss-1", + "actionLabel": "View in KloudMate" + } + ``` + </TabItem> + <TabItem label="SLO burn rate"> + Sent when an [SLO burn-rate alert](../../../reliability/burn-rate-alerts/) starts firing or resolves. `threshold` and the observed burn rates are formatted with a trailing `×` (an infinite rate renders as `∞`); windows render as `5m`, `1h`, etc. + + ```json + { + "subject": "🔥 SLO burn-rate firing: Checkout availability", + "slo_name": "Checkout availability", + "alert_name": "Fast burn", + "severity_tier": "critical", + "threshold": "14.4×", + "short_window": "5m", + "long_window": "1h", + "observed_short_burn": "21.3×", + "observed_long_burn": "16.8×", + "status_line": "Firing since 2026-05-23 10:00:00 UTC", + "actionUrl": "https://app.kloudmate.com/ws-1/slos/slo-1", + "actionLabel": "View SLO" + } + ``` + + On resolve, `subject` becomes `"✅ SLO burn-rate resolved: Checkout availability"` and `status_line` becomes a duration, e.g. `"Fired for 17 minutes"`. `severity_tier` is one of `critical`, `high`, `medium`, or `low`. + </TabItem> + <TabItem label="Synthetic monitor"> + Sent when a [synthetic monitor](../../../synthetic/) goes down or recovers. + + ```json + { + "subject": "🔴 Monitor is DOWN: API health", + "monitor_id": "mon-1", + "name": "API health", + "target": "https://api.example.com/healthz", + "cause": "HTTP 503", + "start_time": "2026-05-23 10:00:00 UTC", + "duration": "", + "actionUrl": "https://app.kloudmate.com/ws-1/synthetics/mon-1", + "actionLabel": "View Details" + } + ``` + + On recovery, `subject` becomes `"🟢 Monitor is UP: API health"`, `duration` is populated (e.g. `"15 minutes"`), and an `end_time` field is added. + </TabItem> + <TabItem label="AI investigation"> + Sent when an [AI investigation](../../../kloudmate-assistant/investigations/) completes. `rootCauseAnalysisText` is plain text, truncated to 8000 characters. + + ```json + { + "investigationId": "inv-1", + "workspaceId": "ws-1", + "title": "API outage investigation", + "subject": "[AI Investigation Completed] API outage (workspace-name / org-name)", + "workspaceName": "workspace-name", + "orgName": "org-name", + "completedAt": "2026-05-23 10:00:00 UTC", + "rootCauseAnalysisText": "Deployment v1.42 introduced a regression in the payments service.", + "actionUrl": "https://app.kloudmate.com/ws-1/assistant/investigations/inv-1", + "actionLabel": "View Investigation" + } + ``` + </TabItem> +</Tabs> + +### KloudMate Incidents + +KloudMate Incidents is the channel type that bridges into [Incident Management](../../../incident-management/). When an alert routes through a KloudMate Incidents channel, KloudMate opens an incident on the linked IM service automatically — no separate webhook plumbing required. + +![Create KloudMate Incidents channel](./images/channel-km-incidents-form.png) + +1. Select **KloudMate Incidents** from the dropdown menu. +2. Pick the IM service you want to bridge into from the **Linked IM service** dropdown. If you don't have any services yet, KloudMate prompts you to create one first. +3. Click **Submit**. + +KloudMate automatically creates two rows: + +- A KloudMate Alert integration under the linked service. +- A notification channel pointing at the integration's signed loop URL. + +KloudMate Incidents channels are **HMAC-signed by default** — the signing secret is generated and stored automatically; you don't need to copy or paste it. The signing secret and linked service are fixed at creation; to change either, delete the channel and create a new one. + +To wire it into the alert flow, add the channel as a destination on a [Routing Rule](../../../alerts/routing-rules/). + ### Jira 1. Select **Jira** from the dropdown menu and click the **Connect Jira** button. @@ -170,20 +355,20 @@ Enter a **Channel Name** and select the following: 4. Click **Submit** to complete the integration. :::info -Jira integration with KloudMate is one-way. Status changes in KloudMate will be reflected in Jira, but status changes made in Jira will not be reflected in KloudMate. +The Jira integration is one-way. Status changes in KloudMate are reflected in Jira, but changes made in Jira are not reflected in KloudMate. ::: -#### Alarm and Issue Details in Jira +#### Alert and Issue Details in Jira -**Alarm details in Jira** +**Alert details in Jira** | KloudMate | Jira | | --- | --- | -| Alarm name | Ticket title | +| Alert name | Ticket title | | KloudMate account name | Ticket title | -| Alarm description | Ticket description | -| Alarm value | Ticket description | -| Regression alarms | Ticket comment | +| Alert description | Ticket description | +| Alert value | Ticket description | +| Regression alerts | Ticket comment | **Issues details in Jira** @@ -197,12 +382,12 @@ Jira integration with KloudMate is one-way. Status changes in KloudMate will be #### Creating a Manual Jira Ticket -You can create a Jira ticket directly from an issue or alarm details page to raise individual tickets that are not assigned to a notification policy. +You can create a Jira ticket directly from an issue or alert details page to raise individual tickets that are not bound to a routing rule. **Prerequisite:** A Jira notification channel must already be configured. -1. Navigate to the **Issue** or **Alarm** details page. -2. Click the **Create in Jira** button. A dialog will open showing all Jira channels integrated with your KloudMate account, along with any previously linked Jira tickets for that issue. +1. Navigate to the **Issue** or **Alert** details page. +2. Click the **Create in Jira** button. A dialog opens showing all Jira channels integrated with your KloudMate account, along with any previously linked Jira tickets for that issue. ![image](./images/create-a-manual-jira-ticket-1.png) ![image](./images/create-a-manual-jira-ticket-1.jpeg) @@ -212,7 +397,7 @@ You can create a Jira ticket directly from an issue or alarm details page to rai #### Unlinking a Jira Ticket -1. On the issue or alarm details page, click the **settings** icon next to the Create in Jira button to open the Jira Tickets panel. +1. On the issue or alert details page, click the **settings** icon next to the Create in Jira button to open the Jira Tickets panel. 2. The panel displays all Jira tickets currently linked to the issue, showing the Ticket ID, Site, Project, and Issue Type. 3. Click the **unlink** icon next to the ticket you want to remove to unlink it from KloudMate. diff --git a/src/content/docs/docs/platform/settings/sso/_meta.json b/src/content/docs/docs/platform/settings/sso/_meta.json new file mode 100644 index 00000000..985eedce --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Single Sign-On (SSO)", + "order": 10 +} diff --git a/src/content/docs/docs/platform/settings/sso/domain-verification.mdx b/src/content/docs/docs/platform/settings/sso/domain-verification.mdx new file mode 100644 index 00000000..96939bad --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/domain-verification.mdx @@ -0,0 +1,77 @@ +--- +title: "Verify your company domain" +description: "Add and DNS-verify your company domain so your team can sign in to KloudMate by typing their work email." +sidebar: + label: "Domain verification" + order: 2 +--- + +Verifying your domain lets an employee type `you@acme.com` on the login page and get routed to your identity provider. Until a domain is verified, email-based sign-in won't reach your IdP — KloudMate has no proof that you control the domain, so it won't hand those users off. + +You verify ownership by publishing a DNS **TXT** record that KloudMate looks up. This is the same kind of check you've done for email or other SaaS tools. + +## Before you start + +- You must be the **organization owner** on a **paid plan**. +- You need access to your domain's DNS records (or someone who can add a TXT record for you). +- You'll usually want at least one [connection](../) added first, so sign-in has somewhere to go once the domain verifies. + +## Add a domain + +1. Open **Settings → Single Sign-On** and find the **Verified domains** card. +2. In **Company domain**, type your domain — for example `acme.com` — and click **Add domain**. +3. The domain appears in the list with a **Pending** badge and the DNS record you need to publish. + +:::caution +You can't verify a public email domain. Addresses like `gmail.com`, `outlook.com`, or `yahoo.com` are rejected with **Public email domains (gmail.com, etc.) can't be used.** Use a domain your company controls. +::: + +## Publish the TXT record + +A pending domain shows the exact record to add. It has two parts: + +- **Host / Name** — `_kloudmate-verify.<your-domain>` (for example `_kloudmate-verify.acme.com`). +- **Value** — a one-time verification token. + +Both fields have a copy button. At your DNS provider, create a new **TXT** record using these values. Leave the TTL at the default. + +![Pending domain with its DNS TXT record](./images/domain-verification-txt.png) + +How the host is entered varies slightly by DNS provider. Many providers append your domain automatically, so you enter only `_kloudmate-verify` as the host or name. If your provider expects the full record name, use `_kloudmate-verify.acme.com`. When in doubt, follow how your provider handles other TXT records. + +## Verify + +Once the record is published, click **Verify** on the domain row. + +DNS changes can take a few minutes — sometimes longer — to propagate. If verification doesn't succeed on the first try, wait a bit and click **Verify** again. On success the badge switches from **Pending** to **Verified** and the verification date is shown. + +## After verification + +A verified domain does two things: + +- **Routes email-based sign-in.** Anyone whose email ends in a verified domain is sent to your IdP when they sign in. +- **Brings members into SSO enforcement.** Members on a verified domain must sign in through SSO from then on — password login is blocked for them. The owner keeps password access as a fallback, and collaborators on *other* domains aren't affected. See [Provisioning & enforcement](../provisioning-and-enforcement/) for the full rules. + +:::note +One organization per domain. A domain can be verified by only one KloudMate organization — first to verify wins. If you try to verify a domain another org already claimed, you'll see **This domain is already verified by another organization.** Contact support if that domain genuinely belongs to you. +::: + +## Remove a domain + +Use the delete (trash) icon on a domain row to remove it. KloudMate asks you to confirm, because members on that domain will no longer be able to sign in by typing their work email. Removing a domain doesn't delete your connections — it only stops email-based routing for that domain. + +## Troubleshooting + +| What you see | Likely cause | Fix | +| --- | --- | --- | +| **TXT record not found or didn't match** | The record isn't published yet, has the wrong host/value, or DNS hasn't propagated | Re-check the **Host / Name** and **Value** against the row, wait a few minutes, then click **Verify** again | +| **Public email domains (gmail.com, etc.) can't be used** | You entered a shared public domain | Use a domain your company owns | +| **This domain is already verified by another organization** | Another org claimed it first | Contact support if the domain is yours | + +For sign-in-time errors, see the full [Troubleshooting](../troubleshooting/) page. + +## Related + +- [SSO overview & setup](../) +- [Provisioning & enforcement](../provisioning-and-enforcement/) +- [Troubleshooting](../troubleshooting/) diff --git a/src/content/docs/docs/platform/settings/sso/images/domain-verification-txt.png b/src/content/docs/docs/platform/settings/sso/images/domain-verification-txt.png new file mode 100644 index 00000000..ae80c136 Binary files /dev/null and b/src/content/docs/docs/platform/settings/sso/images/domain-verification-txt.png differ diff --git a/src/content/docs/docs/platform/settings/sso/images/sso-settings-page.png b/src/content/docs/docs/platform/settings/sso/images/sso-settings-page.png new file mode 100644 index 00000000..e7d78471 Binary files /dev/null and b/src/content/docs/docs/platform/settings/sso/images/sso-settings-page.png differ diff --git a/src/content/docs/docs/platform/settings/sso/images/user-provisioning-card.png b/src/content/docs/docs/platform/settings/sso/images/user-provisioning-card.png new file mode 100644 index 00000000..fb09fcfd Binary files /dev/null and b/src/content/docs/docs/platform/settings/sso/images/user-provisioning-card.png differ diff --git a/src/content/docs/docs/platform/settings/sso/index.mdx b/src/content/docs/docs/platform/settings/sso/index.mdx new file mode 100644 index 00000000..d1f399f4 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/index.mdx @@ -0,0 +1,112 @@ +--- +title: "Single Sign-On (SSO)" +description: "Connect your company identity provider so your team signs in to KloudMate through SAML 2.0 or OIDC." +sidebar: + label: "Overview" + order: 1 +--- + +Single Sign-On lets your team sign in to KloudMate through your own identity provider (IdP) — Okta, Microsoft Entra ID, Google, and others — instead of a separate KloudMate password. You connect the IdP once, verify your company domain, and from then on your team signs in to KloudMate by typing their work email. + +KloudMate supports two protocols: + +- **SAML 2.0** — the long-standing enterprise standard, supported by every major IdP. +- **OIDC (OpenID Connect)** — a lighter, JSON/OAuth-based flow. When your IdP offers both, OIDC is usually quicker to set up. + +## Before you start + +- You must be the **organization owner**. Other roles can't see or change SSO settings. +- Your organization must be on a **paid plan**. +- You'll need admin access to your IdP (to create an app/connection) and access to your domain's **DNS** records (to verify the domain). + +:::note +The owner always keeps password access as a fallback, even after SSO is enforced. That's deliberate — it stops you locking yourself out if the IdP connection breaks. See [Provisioning & enforcement](./provisioning-and-enforcement/). +::: + +## Where to find it + +Open **Settings → Single Sign-On**. The item sits in the organization menu and only appears for the owner. + +The page is split into four cards, which also map to the setup order: + +1. **Connections** — the IdP(s) your team signs in through. +2. **Service provider details** — the KloudMate values you give your IdP. +3. **User provisioning** — whether accounts are created automatically or by invite. +4. **Verified domains** — the company domains that route email-based sign-in. + +![KloudMate Single Sign-On settings page](./images/sso-settings-page.png) + +## The setup at a glance + +Four steps take you from nothing to working SSO: + +1. **Add a connection** for your IdP (SAML or OIDC). This is where you hand KloudMate the IdP's details. +2. **Give your IdP the KloudMate values** — the ACS URL (SAML) or redirect URI (OIDC), or upload the downloaded SP metadata. +3. **Verify your company domain** so employees can sign in by typing their work email. +4. **Choose a provisioning mode** — invite-only or just-in-time. + +The first two steps happen together: you create the app in your IdP, paste KloudMate's values into it, then copy the IdP's values back into KloudMate. The [provider guides](./providers/) walk through this for each IdP. + +## Add a connection + +In the **Connections** card, click **Add connection**. Give it a **Display name** (a label you choose, like `Okta` or `Azure AD`), then pick the **Connection type**: **SAML 2.0** or **OIDC**. + +### SAML 2.0 + +For SAML you choose how to supply the IdP's details with the **Configuration** control: + +- **Paste metadata XML** (preferred) — paste the IdP's metadata document into **IdP metadata XML**. This carries the IdP's SSO URL, entity ID, and signing certificate in one block, so there's nothing else to copy. +- **Enter manually** — when you don't have the metadata, fill in three fields: + - **SSO URL (entry point)** — the IdP's SAML sign-in URL. + - **Issuer (IdP entity ID)** — the IdP's entity ID. + - **IdP signing certificate (X.509)** — the IdP's public signing certificate, including the `-----BEGIN CERTIFICATE-----` lines. + +KloudMate reads the user's email from the SAML **NameID** (format `emailAddress`) or from an attribute named `email`. If your IdP sends those under non-standard names, expand the optional **attribute mapping** and set the **Email attribute** (and **Name attribute**) to match. + +### OIDC + +For OIDC, enter: + +- **Discovery URL** — your IdP's `.well-known/openid-configuration` endpoint. It must be a public `https://` URL. +- **Client ID** and **Client secret** — from a **confidential** web-app client you create in the IdP. +- **Scopes** — defaults to `openid email profile`. Leave it unless your IdP needs more. + +Click **Add connection** to save. The connection appears in the table with its name, type, and last-updated time. + +## Give your IdP the KloudMate values + +Your IdP needs to know where to send users back and who it's talking to. Open the **Service provider details** card to get these. + +For SAML, the card shows a read-only **ACS URL (Assertion Consumer Service)** with a copy button, and a **Download SP metadata** button. You have two options: + +- **Upload SP metadata** — download `kloudmate-sp-metadata.xml` and import it into any IdP that accepts SP metadata. This fills in the ACS URL and entity ID for you. +- **Enter values by hand** — copy the **ACS URL** from the screen, and read the **SP Entity ID / Audience** from the `entityID` attribute inside the downloaded metadata file. + +The canonical production values are: + +| What your IdP asks for | Value to use | +| --- | --- | +| SAML ACS / Reply / Recipient / Destination URL | `https://api.kloudmate.com/sso/saml` | +| SAML SP Entity ID / Audience URI | the `entityID` in the downloaded SP metadata | +| SAML NameID format | `emailAddress` (email in the NameID, or an `email` attribute) | +| OIDC Redirect / Callback / Sign-in redirect URI | `https://api.kloudmate.com/sso/oidc` | +| OIDC scopes | `openid email profile` | + +## Edit or remove a connection + +Use the **⋯** menu on a connection row to **Edit** or **Delete** it. + +Secrets are write-only: when you edit, the client secret and signing certificate are never shown back to you. Leave those fields blank to keep the stored value, or fill them in to replace it. Deleting a connection stops anyone from signing in through it, so don't remove the connection your team is actively using. + +## Next steps + +- [Verify your company domain](./domain-verification/) — required for email-based sign-in. +- [Provisioning & enforcement](./provisioning-and-enforcement/) — control who gets an account and who's forced through SSO. +- [Provider guides](./providers/) — step-by-step setup for Okta, Entra ID, Google, Auth0, OneLogin, and generic IdPs. +- [Troubleshooting](./troubleshooting/) — what each sign-in error means and how to fix it. + +## Related + +- [Users & Permissions](../users-permissions/) +- [Profile & Security](../profile-security/) +- [Subscription & Billing](../subscription-billing/) diff --git a/src/content/docs/docs/platform/settings/sso/providers/_meta.json b/src/content/docs/docs/platform/settings/sso/providers/_meta.json new file mode 100644 index 00000000..9927a6f8 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "Provider guides", + "order": 4 +} diff --git a/src/content/docs/docs/platform/settings/sso/providers/auth0.mdx b/src/content/docs/docs/platform/settings/sso/providers/auth0.mdx new file mode 100644 index 00000000..821f3d64 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/auth0.mdx @@ -0,0 +1,93 @@ +--- +title: "Set up SSO with Auth0" +description: "Connect Auth0 to KloudMate using OIDC (Regular Web App) or the SAML2 Web App addon." +sidebar: + label: "Auth0" + order: 5 +--- + +This guide connects Auth0 to KloudMate. Auth0 supports both OIDC and SAML. **OIDC is the simpler setup** and the recommended path. + +## Before you start + +- You're the KloudMate **organization owner** on a **paid plan**. +- You can create applications in the Auth0 Dashboard. +- Copy KloudMate's values first from the **Service provider details** card. + +## OIDC (recommended) + +### 1. Create the application + +1. In the Auth0 Dashboard, go to **Applications → Applications → Create Application**. +2. Choose **Regular Web Applications** and create it. +3. On the **Settings** tab, under **Allowed Callback URLs**, add: + + ```text + https://api.kloudmate.com/sso/oidc + ``` + +4. Save changes. + +### 2. Copy Auth0's values into KloudMate + +From the application's **Settings**, copy the **Client ID** and **Client Secret**. Your Auth0 discovery URL is: + +```text +https://<your-tenant>.auth0.com/.well-known/openid-configuration +``` + +Replace `<your-tenant>` with your Auth0 tenant (use your custom domain if you've set one). + +In KloudMate, open **Connections → Add connection**, choose **OIDC**, and fill in: + +- **Display name** — `Auth0` +- **Discovery URL** — the URL above +- **Client ID** and **Client secret** — from Auth0 +- **Scopes** — leave as `openid email profile` + +Click **Add connection**. + +### Email mapping + +With the default `email` scope, Auth0 returns the user's email in the ID token. No extra mapping is needed unless you have customized the token with a rule or action that drops the standard claims. + +## SAML + +Auth0 exposes SAML through the **SAML2 Web App** addon on an application. + +### 1. Enable and configure the addon + +1. Open your Auth0 application, go to the **Addons** tab, and enable **SAML2 Web App**. +2. In **Application Callback URL**, enter your KloudMate **ACS URL**: + + ```text + https://api.kloudmate.com/sso/saml + ``` + +3. In the **Settings** JSON for the addon, set the audience to your KloudMate **SP Entity ID** (the `entityID` from the downloaded SP metadata): + + ```json + { + "audience": "YOUR_KLOUDMATE_SP_ENTITY_ID" + } + ``` + +4. Save. + +### 2. Copy Auth0's IdP metadata into KloudMate + +1. In the SAML2 Web App addon's **Usage** tab, copy the **Identity Provider Metadata** URL or download the metadata XML. +2. In KloudMate, open **Connections → Add connection**, choose **SAML 2.0**, keep **Paste metadata XML**, and paste the metadata into **IdP metadata XML**. +3. Set **Display name** and click **Add connection**. + +### Email mapping + +Auth0's default SAML mapping puts the email in the NameID, which KloudMate reads. If you have overridden the mappings in the addon settings, make sure the email is sent in the NameID or an `email` attribute. + +## Finish up + +1. [Verify your company domain](../../domain-verification/). +2. Choose a [provisioning mode](../../provisioning-and-enforcement/). +3. Test sign-in with a work email on a verified domain via **Sign in with SSO**. + +Hitting an error? See [Troubleshooting](../../troubleshooting/). diff --git a/src/content/docs/docs/platform/settings/sso/providers/entra-id.mdx b/src/content/docs/docs/platform/settings/sso/providers/entra-id.mdx new file mode 100644 index 00000000..a012ddd5 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/entra-id.mdx @@ -0,0 +1,93 @@ +--- +title: "Set up SSO with Microsoft Entra ID" +description: "Connect Microsoft Entra ID (Azure AD) to KloudMate using OIDC or SAML, with the tenant-specific discovery URL and email claim notes." +sidebar: + label: "Microsoft Entra ID" + order: 3 +--- + +This guide connects Microsoft Entra ID (formerly Azure AD) to KloudMate. Entra supports both OIDC and SAML. **OIDC is the simpler setup**, but it has one easy-to-miss requirement — read the discovery-URL note below before you start. + +## Before you start + +- You're the KloudMate **organization owner** on a **paid plan**. +- You can create app registrations / enterprise apps in the Microsoft Entra admin center. +- Copy KloudMate's values first from the **Service provider details** card. + +:::caution +**Use your tenant-specific discovery URL, not `common`.** With `common` or `organizations`, sign-in fails with an issuer mismatch (`OidcCallbackFailed`). The correct form is: + +```text +https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration +``` + +Replace `<tenant-id>` with your directory (tenant) ID from the Entra **Overview** page. +::: + +## OIDC (recommended) + +### 1. Register the app + +1. In the Microsoft Entra admin center, go to **Identity → Applications → App registrations → New registration**. +2. Name it (for example `KloudMate`). +3. Under **Redirect URI**, choose **Web** and enter: + + ```text + https://api.kloudmate.com/sso/oidc + ``` + +4. Register the app, then copy the **Application (client) ID** and **Directory (tenant) ID** from the **Overview** page. + +### 2. Create a client secret + +1. Open **Certificates & secrets → Client secrets → New client secret**. +2. Add a description and expiry, then **Add**. +3. Copy the secret **Value** immediately — Entra only shows it once. + +### 3. Copy the values into KloudMate + +In KloudMate, open **Connections → Add connection**, choose **OIDC**, and fill in: + +- **Display name** — `Azure AD` (or `Entra ID`) +- **Discovery URL** — the **tenant-specific** URL from the caution above, with your tenant ID +- **Client ID** — the Application (client) ID +- **Client secret** — the secret **Value** you copied +- **Scopes** — leave as `openid email profile` + +Click **Add connection**. + +### Email claim note + +Entra often omits the `email` claim. KloudMate falls back to `preferred_username` / `upn`, which works for most users — to ensure reliability, make sure each user has a mailbox, or add the optional **email** claim to the app's token configuration: + +1. Open **Token configuration → Add optional claim**. +2. Choose **ID** as the token type, select **email**, and save. +3. If prompted, grant the related Microsoft Graph permission. + +## SAML + +### 1. Create the enterprise app + +1. Go to **Identity → Applications → Enterprise applications → New application → Create your own application**. +2. Choose **Integrate any other application you don't find in the gallery**, name it, and create it. +3. Open **Single sign-on → SAML**, and under **Basic SAML Configuration** set: + - **Identifier (Entity ID)** — your KloudMate **SP Entity ID** (the `entityID` from the downloaded SP metadata) + - **Reply URL (Assertion Consumer Service URL)** — your KloudMate **ACS URL**, `https://api.kloudmate.com/sso/saml` + +### 2. Copy Entra's metadata into KloudMate + +1. In the SAML setup page, download the **Federation Metadata XML** (or copy the **App Federation Metadata Url**). +2. In KloudMate, open **Connections → Add connection**, choose **SAML 2.0**, keep **Paste metadata XML**, and paste the Federation Metadata XML into **IdP metadata XML**. +3. Set **Display name** and click **Add connection**. + +### Email mapping + +In the enterprise app, open **Attributes & Claims** and confirm the **Unique User Identifier (Name ID)** is the user's email (for example `user.mail` or `user.userprincipalname`), in `emailAddress` format. If your users' UPN isn't their email, map an `email` attribute to `user.mail` and set the corresponding **Email attribute** in KloudMate's manual SAML form. + +## Finish up + +1. [Verify your company domain](../../domain-verification/). +2. Choose a [provisioning mode](../../provisioning-and-enforcement/). +3. Test sign-in with a work email on a verified domain via **Sign in with SSO**. + +For issuer or email errors, see [Troubleshooting](../../troubleshooting/). diff --git a/src/content/docs/docs/platform/settings/sso/providers/generic.mdx b/src/content/docs/docs/platform/settings/sso/providers/generic.mdx new file mode 100644 index 00000000..cff64ba3 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/generic.mdx @@ -0,0 +1,92 @@ +--- +title: "Generic SAML & OIDC setup" +description: "Field-by-field mapping to connect any standards-compliant SAML or OIDC identity provider to KloudMate, including self-hosted IdPs." +sidebar: + label: "Generic SAML & OIDC" + order: 7 +--- + +Don't see your identity provider in the other guides? Any standards-compliant SAML 2.0 or OIDC provider works. This page maps KloudMate's fields to the generic terms your IdP uses, so you can wire up the two sides regardless of vendor wording. + +Copy KloudMate's values from the **Service provider details** card before you start — the host depends on your environment, so don't assume `api.kloudmate.com`. + +## Generic OIDC + +OIDC needs a **confidential** client (one that can hold a secret) and a discovery endpoint. + +### What to create in your IdP + +A web/confidential application (authorization-code flow) with this redirect URI: + +```text +https://api.kloudmate.com/sso/oidc +``` + +### Field mapping + +| KloudMate field | Your IdP calls it | Notes | +| --- | --- | --- | +| **Discovery URL** | OpenID configuration / well-known endpoint | Must be a public `https://` URL ending in `/.well-known/openid-configuration` | +| **Client ID** | Client ID / Application ID | — | +| **Client secret** | Client secret | From a confidential client | +| **Scopes** | Scopes | Defaults to `openid email profile`; keep `email` so KloudMate gets the address | + +### Email claim + +KloudMate reads the user's email from the `email` claim in the ID token. Make sure the `email` scope is granted and your IdP includes that claim. If your IdP only returns email under a different claim, configure it to emit a standard `email` claim. + +## Generic SAML + +SAML can be configured two ways in KloudMate: paste the IdP's metadata XML (preferred), or enter the three core values by hand. + +### What to create in your IdP + +A SAML 2.0 application / relying party with: + +| Your IdP's field | Value to use | +| --- | --- | +| ACS / Reply / Recipient / Destination URL | `https://api.kloudmate.com/sso/saml` | +| Audience / SP Entity ID | the `entityID` in KloudMate's downloaded SP metadata | +| NameID format | `emailAddress` | + +Many IdPs accept KloudMate's **SP metadata** directly — download `kloudmate-sp-metadata.xml` from the **Service provider details** card and import it to fill the ACS URL and entity ID automatically. + +### Field mapping (manual entry) + +If your IdP can't export metadata, choose **Enter manually** in the connection dialog and map: + +| KloudMate field | Your IdP calls it | +| --- | --- | +| **SSO URL (entry point)** | SAML SSO / sign-in endpoint (HTTP-Redirect or HTTP-POST) | +| **Issuer (IdP entity ID)** | IdP entity ID / issuer | +| **IdP signing certificate (X.509)** | Token-signing / X.509 certificate (PEM, including the `BEGIN/END CERTIFICATE` lines) | + +### Email and attribute mapping + +KloudMate reads the email from the **NameID** (`emailAddress` format) or from an attribute named `email`. If your IdP sends email or name under non-standard attribute names, expand the optional **attribute mapping** in the connection dialog and set: + +- **Email attribute** — the assertion attribute that holds the email +- **Name attribute** — the assertion attribute that holds the display name + +## Self-hosted identity providers + +Self-hosted IdPs such as Keycloak work the same way, with one hard requirement: + +:::caution +The discovery URL (OIDC) or metadata/SSO endpoint (SAML) must be reachable over **public HTTPS**. KloudMate's servers fetch these directly. An IdP that's only reachable inside your private network, on an internal hostname, or over plain HTTP can't be connected. +::: + +For Keycloak specifically: + +- **OIDC discovery URL**: `https://<keycloak-host>/realms/<realm>/.well-known/openid-configuration` +- **SAML metadata**: `https://<keycloak-host>/realms/<realm>/protocol/saml/descriptor` + +Create a confidential OIDC client (or a SAML client) in your realm, set the redirect URI / ACS URL to KloudMate's values, and make sure the realm issues the user's email. + +## Finish up + +1. [Verify your company domain](../../domain-verification/). +2. Choose a [provisioning mode](../../provisioning-and-enforcement/). +3. Test sign-in with a work email on a verified domain via **Sign in with SSO**. + +Hitting an error? See [Troubleshooting](../../troubleshooting/) — `InvalidOidcConnection` / `UnsafeUrl` usually means the discovery URL isn't publicly reachable over HTTPS. diff --git a/src/content/docs/docs/platform/settings/sso/providers/google.mdx b/src/content/docs/docs/platform/settings/sso/providers/google.mdx new file mode 100644 index 00000000..338f28f0 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/google.mdx @@ -0,0 +1,72 @@ +--- +title: "Set up SSO with Google" +description: "Connect Google (Google Cloud OAuth) to KloudMate using OIDC, with notes on restricting sign-in to your Workspace domain." +sidebar: + label: "Google" + order: 4 +--- + +This guide connects Google to KloudMate using OIDC, via OAuth credentials you create in Google Cloud. Google is OIDC-only here — there's no SAML variant in this guide. + +:::caution +**Plain "Sign in with Google" is not restricted to your company** — any Google account can complete it. To limit sign-in to your organization and use email-based routing, you need a **Google Workspace** (custom-domain) account and you must verify that domain in KloudMate. A `gmail.com` domain can't be verified. +::: + +## Before you start + +- You're the KloudMate **organization owner** on a **paid plan**. +- You can create credentials in a Google Cloud project. +- You use **Google Workspace** with your own domain (so you can verify it later). + +## 1. Configure the OAuth consent screen + +1. In the [Google Cloud console](https://console.cloud.google.com/), pick or create a project. +2. Go to **APIs & Services → OAuth consent screen**. +3. Choose **Internal** if you want only your Workspace users (recommended), complete the app name and support email, and save. + +## 2. Create OAuth credentials + +1. Go to **APIs & Services → Credentials → Create credentials → OAuth client ID**. +2. Choose **Web application**. +3. Under **Authorized redirect URIs**, add: + + ```text + https://api.kloudmate.com/sso/oidc + ``` + +4. Create the client, then copy the **Client ID** and **Client secret**. + +## 3. Add the connection in KloudMate + +Open **Connections → Add connection**, choose **OIDC**, and fill in: + +- **Display name** — `Google` +- **Discovery URL**: + + ```text + https://accounts.google.com/.well-known/openid-configuration + ``` + +- **Client ID** and **Client secret** — from the OAuth client +- **Scopes** — leave as `openid email profile` + +Click **Add connection**. + +### Email mapping + +With the `email` scope (included by default), Google returns the user's email in the ID token, which is exactly what KloudMate reads — no extra mapping needed. + +## 4. Restrict to your domain + +Because Google itself doesn't gate this flow to one company, KloudMate's domain verification does the gating: + +1. [Verify your company domain](../../domain-verification/) — your Workspace domain, for example `acme.com`. +2. Only emails on that verified domain route through this connection when someone signs in by email. + +Then choose a [provisioning mode](../../provisioning-and-enforcement/). With **invite-only**, even a user who authenticates with Google must already be invited — recommended given Google's open consent flow. + +## Finish up + +Test sign-in: on the login page, type a Workspace email on your verified domain and click **Sign in with SSO**. + +Hitting an error? See [Troubleshooting](../../troubleshooting/). diff --git a/src/content/docs/docs/platform/settings/sso/providers/index.mdx b/src/content/docs/docs/platform/settings/sso/providers/index.mdx new file mode 100644 index 00000000..ca91aaa4 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/index.mdx @@ -0,0 +1,42 @@ +--- +title: "SSO provider guides" +description: "Step-by-step SSO setup for Okta, Microsoft Entra ID, Google, Auth0, OneLogin, and any generic SAML or OIDC provider." +sidebar: + label: "Overview" + order: 1 +--- + +These guides walk through connecting a specific identity provider to KloudMate end to end: what to create in the IdP, where to paste KloudMate's values, and what to copy back. Each one assumes you've read the [SSO overview](../) and that you're the organization owner on a paid plan. + +## Pick your provider + +- [Okta](./okta/) — OIDC and SAML +- [Microsoft Entra ID (Azure AD)](./entra-id/) — OIDC and SAML +- [Google](./google/) — OIDC via Google Cloud +- [Auth0](./auth0/) — OIDC and SAML +- [OneLogin](./onelogin/) — SAML +- [Generic SAML or OIDC](./generic/) — the field-by-field mapping for any compliant IdP, plus self-hosted notes + +## SAML or OIDC? + +When your IdP supports both, **OIDC is usually the simpler path** — you exchange a discovery URL and a client ID/secret instead of wrangling XML metadata and certificates. Use SAML when your IdP only offers SAML, or when your security team standardizes on it. + +## The KloudMate values you'll need + +Every guide pastes the same handful of values into the IdP. Copy the **ACS URL** from your own **Service provider details** card rather than assuming the host — it depends on your environment. + +| What your IdP asks for | Value to use | +| --- | --- | +| SAML ACS / Reply / Recipient / Destination URL | `https://api.kloudmate.com/sso/saml` | +| SAML SP Entity ID / Audience URI | the `entityID` in the downloaded SP metadata | +| SAML NameID format | `emailAddress` (email in the NameID, or an `email` attribute) | +| OIDC Redirect / Callback / Sign-in redirect URI | `https://api.kloudmate.com/sso/oidc` | +| OIDC scopes | `openid email profile` | + +After you finish a provider guide, [verify your company domain](../domain-verification/) so your team can sign in by typing their work email, then choose a [provisioning mode](../provisioning-and-enforcement/). + +## Related + +- [SSO overview & setup](../) +- [Verify your company domain](../domain-verification/) +- [Troubleshooting](../troubleshooting/) diff --git a/src/content/docs/docs/platform/settings/sso/providers/okta.mdx b/src/content/docs/docs/platform/settings/sso/providers/okta.mdx new file mode 100644 index 00000000..ddc3f104 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/okta.mdx @@ -0,0 +1,85 @@ +--- +title: "Set up SSO with Okta" +description: "Connect Okta to KloudMate using OIDC or SAML, including app creation, redirect URIs, and email claim mapping." +sidebar: + label: "Okta" + order: 2 +--- + +This guide connects Okta to KloudMate. Okta supports both OIDC and SAML; **OIDC is the simpler setup**, so start there unless your team standardizes on SAML. + +You will switch between two browser tabs: the Okta Admin Console and KloudMate's **Settings → Single Sign-On** page. Keep both open. + +## Before you start + +- You're the KloudMate **organization owner** on a **paid plan**. +- You're an admin in the Okta Admin Console. +- Grab KloudMate's values first: open the **Service provider details** card and copy the **ACS URL** (SAML), or note the OIDC redirect URI `https://api.kloudmate.com/sso/oidc`. + +## OIDC (recommended) + +### 1. Create the app in Okta + +1. In the Okta Admin Console, go to **Applications → Applications → Create App Integration**. +2. Choose **OIDC - OpenID Connect**, then **Web Application**, and click **Next**. +3. Under **Grant types**, keep **Authorization Code** selected. +4. In **Sign-in redirect URIs**, enter: + + ```text + https://api.kloudmate.com/sso/oidc + ``` + +5. Assign the app to the people or groups who should have KloudMate access, then **Save**. + +### 2. Copy Okta's values into KloudMate + +From the app's **General** tab, copy the **Client ID** and **Client secret**. Okta's OIDC discovery URL is: + +```text +https://<your-okta-domain>/.well-known/openid-configuration +``` + +Replace `<your-okta-domain>` with your org's Okta domain (for example `dev-12345.okta.com`). + +In KloudMate, open **Connections → Add connection**, choose **OIDC**, and fill in: + +- **Display name** — `Okta` +- **Discovery URL** — the URL above +- **Client ID** and **Client secret** — from Okta +- **Scopes** — leave as `openid email profile` + +Click **Add connection**. + +:::note +If your Okta app has **Require PKCE as additional verification** turned on, sign-in still works — KloudMate sends a PKCE challenge automatically. +::: + +## SAML + +### 1. Create the SAML app in Okta + +1. Go to **Applications → Applications → Create App Integration**, choose **SAML 2.0**, and click **Next**. +2. Give the app a name, then on the **Configure SAML** step set: + - **Single sign-on URL** — your KloudMate **ACS URL**, `https://api.kloudmate.com/sso/saml` + - **Audience URI (SP Entity ID)** — your KloudMate **SP Entity ID** (the `entityID` from the downloaded SP metadata) + - **Name ID format** — `EmailAddress` + - **Application username** — `Email` +3. Finish the wizard and **Save**. + +### 2. Copy Okta's metadata into KloudMate + +1. On the app's **Sign On** tab, find the **SAML Setup** section and download or view the **Identity Provider metadata**. +2. In KloudMate, open **Connections → Add connection**, choose **SAML 2.0**, keep **Paste metadata XML**, and paste the Okta metadata into **IdP metadata XML**. +3. Set **Display name** to `Okta` and click **Add connection**. + +### Email mapping + +Okta sends the user's email in the NameID when you set **Name ID format** to `EmailAddress` and **Application username** to `Email`, which is all KloudMate needs. If you have customized attribute statements, make sure email is sent in the NameID or in an attribute named `email`. + +## Finish up + +1. [Verify your company domain](../../domain-verification/) so employees can sign in by email. +2. Choose a [provisioning mode](../../provisioning-and-enforcement/). +3. Test sign-in: on the login page, type a work email on a verified domain and click **Sign in with SSO**. + +Hitting an error? See [Troubleshooting](../../troubleshooting/). diff --git a/src/content/docs/docs/platform/settings/sso/providers/onelogin.mdx b/src/content/docs/docs/platform/settings/sso/providers/onelogin.mdx new file mode 100644 index 00000000..d337bf86 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/providers/onelogin.mdx @@ -0,0 +1,53 @@ +--- +title: "Set up SSO with OneLogin" +description: "Connect OneLogin to KloudMate using a SAML Custom Connector, including ACS, audience, and email attribute mapping." +sidebar: + label: "OneLogin" + order: 6 +--- + +This guide connects OneLogin to KloudMate using SAML, via OneLogin's **SAML Custom Connector**. + +## Before you start + +- You're the KloudMate **organization owner** on a **paid plan**. +- You're an admin in OneLogin. +- Copy KloudMate's values first from the **Service provider details** card — you'll need the **ACS URL** and the **SP Entity ID** (the `entityID` in the downloaded SP metadata). + +## 1. Create the SAML app + +1. In the OneLogin admin portal, go to **Applications → Applications → Add App**. +2. Search for **SAML Custom Connector (Advanced)** and add it. +3. Give it a display name like `KloudMate` and save. + +## 2. Configure the SAML settings + +On the app's **Configuration** tab, set: + +- **ACS (Consumer) URL** — your KloudMate **ACS URL**, `https://api.kloudmate.com/sso/saml` +- **ACS (Consumer) URL Validator** — a regex that matches the ACS URL (OneLogin requires one; escape the dots, for example `^https:\/\/api\.kloudmate\.com\/sso\/saml$`) +- **Audience (EntityID)** — your KloudMate **SP Entity ID** +- **Recipient** — the same as the ACS URL +- **NameID format** — `Email` + +Save the configuration. + +## 3. Map the email attribute + +On the **Parameters** tab, confirm the NameID (or an attribute named `email`) is set to the user's **Email**. KloudMate reads the email from the NameID or an `email` attribute, so one of those must carry it. + +## 4. Copy OneLogin's metadata into KloudMate + +1. On the app's **More Actions** menu, choose **SAML Metadata** to download the IdP metadata XML (or copy the **Issuer URL** / metadata URL from the **SSO** tab). +2. In KloudMate, open **Connections → Add connection**, choose **SAML 2.0**, keep **Paste metadata XML**, and paste the metadata into **IdP metadata XML**. +3. Set **Display name** to `OneLogin` and click **Add connection**. + +To configure manually instead of pasting metadata, switch the dialog to **Enter manually** and copy OneLogin's **SAML 2.0 Endpoint (HTTP)** into **SSO URL (entry point)**, its **Issuer URL** into **Issuer (IdP entity ID)**, and its **X.509 Certificate** into **IdP signing certificate**. + +## Finish up + +1. [Verify your company domain](../../domain-verification/). +2. Choose a [provisioning mode](../../provisioning-and-enforcement/). +3. Test sign-in with a work email on a verified domain via **Sign in with SSO**. + +Hitting an error? See [Troubleshooting](../../troubleshooting/). diff --git a/src/content/docs/docs/platform/settings/sso/provisioning-and-enforcement.mdx b/src/content/docs/docs/platform/settings/sso/provisioning-and-enforcement.mdx new file mode 100644 index 00000000..6af90906 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/provisioning-and-enforcement.mdx @@ -0,0 +1,48 @@ +--- +title: "Provisioning & enforcement" +description: "Control who gets a KloudMate account through SSO and who is required to sign in through your identity provider." +sidebar: + label: "Provisioning & enforcement" + order: 3 +--- + +Two settings decide who can sign in through SSO and how their accounts come to exist. **Provisioning** controls whether accounts are created automatically. **Enforcement** controls who is *required* to use SSO instead of a password. This page covers both, and the safeguards that keep you from locking yourself out. + +## Choose a provisioning mode + +In the **User provisioning** card on **Settings → Single Sign-On**, pick one of two modes. The setting takes effect as soon as you select it. + +- **Invite only** *(default)* — only people you've already invited to the organization can sign in through your IdP. Authenticating with the IdP isn't enough on its own; the person also has to have a KloudMate invite. +- **Just-in-time (JIT)** — KloudMate creates an account the first time someone signs in through your IdP. Anyone your IdP lets through becomes a member automatically, without a separate invite. + +![User provisioning options](./images/user-provisioning-card.png) + +### Which one should you pick? + +- Use **Invite only** when you want to control the member list yourself and treat the IdP purely as the login method. A user who authenticates but was never invited is turned away with **Your account isn't set up for this org — ask an admin to invite you.** +- Use **JIT** when your IdP app is already scoped to the right group of people, and you'd rather not invite each person by hand. Everyone in that IdP group can sign in and is added on first login. + +You can switch modes at any time. Switching to JIT doesn't retroactively add anyone — accounts are created as people sign in. + +:::note +JIT adds the person to your **organization**, but it doesn't grant any **workspace** access. Until you assign their workspace permissions, they can sign in but won't see any workspace. As the owner, give them access from [Users & Permissions](../../users-permissions/). +::: + +## How enforcement works + +Once SSO is set up, enforcement is **scoped to your verified domains**. The rules: + +- **Members on a verified domain must use SSO.** If someone whose email is on a verified domain tries to sign in with a password, KloudMate blocks the password path and sends them to your IdP instead. They never see an "invalid password" error — they're sent straight to your IdP. +- **The owner keeps password access.** As the owner, you can always sign in with your password, even after enforcement is on. This is a deliberate lockout-safety fallback: if your IdP connection breaks, you can still get in and fix it. +- **External and guest collaborators are not forced through your IdP.** A collaborator whose email is on a different domain — a contractor on their own company's domain, for example — keeps normal password login. Your IdP only governs your own domains. + +:::caution +Before you rely on enforcement, confirm you can complete a full SSO sign-in with a test account on your domain. Keep your owner password available until you've verified the round trip works end to end — that password is your way back in if the IdP connection has a problem. +::: + +## Related + +- [SSO overview & setup](../) +- [Verify your company domain](../domain-verification/) +- [Users & Permissions](../../users-permissions/) +- [Troubleshooting](../troubleshooting/) diff --git a/src/content/docs/docs/platform/settings/sso/troubleshooting.mdx b/src/content/docs/docs/platform/settings/sso/troubleshooting.mdx new file mode 100644 index 00000000..a5a213f4 --- /dev/null +++ b/src/content/docs/docs/platform/settings/sso/troubleshooting.mdx @@ -0,0 +1,62 @@ +--- +title: "Troubleshooting SSO" +description: "Decode KloudMate's SSO sign-in errors — what each code means, the likely cause, and how to fix it." +sidebar: + label: "Troubleshooting" + order: 5 +--- + +When an SSO sign-in fails, KloudMate lands the user on a branded **We couldn't sign you in** screen with a short, plain-language reason. If the identity provider returned a raw detail (e.g. `invalid_client`), it is available under a **Technical details** expander — open it when debugging the setup. A **Back to login** button returns to the start. + +This page lists the errors you might hit, grouped by where they happen, with the most likely cause and the fix. + +## Sign-in (callback) errors + +These appear on the callback screen after the user comes back from the IdP. + +| Error | What it means | How to fix it | +| --- | --- | --- | +| **SSO isn't set up for this organization or email** (`SSONotConfigured`) | No SSO connection or verified domain matches that email | Confirm a [connection](../) exists and the user's domain is [verified](../domain-verification/) | +| **Couldn't verify your identity provider's response** (`OidcCallbackFailed`) | The OIDC token or ID token failed validation | Check the **Client secret** (a wrong one shows `invalid_client` in the details), confirm the redirect URI `https://api.kloudmate.com/sso/oidc` is registered in the IdP, and for Entra ID use the **tenant-specific** discovery URL, not `common`. Large clock skew on either side can also cause this | +| **Couldn't verify your identity provider's response** (`SamlValidationFailed`) | The SAML assertion's signature, audience, or destination didn't check out | Re-import the IdP signing certificate (or fresh metadata), make sure the IdP **Audience** equals your **SP Entity ID**, and that the **Recipient/Destination** equals your **ACS URL** (`https://api.kloudmate.com/sso/saml`) | +| **Your identity provider didn't return an email address** (`OidcMissingEmail` / `SamlMissingEmail`) | The IdP didn't send an email claim | Map the email claim (OIDC) or send email in the **NameID** or an `email` attribute (SAML). For Entra ID, ensure the user has a mailbox or map the optional `email` claim — see [the Entra guide](../providers/entra-id/) | +| **Your account isn't set up for this org — ask an admin to invite you** (`UserNotProvisioned` / `UserNotInOrg`) | Provisioning is **invite-only** and this person wasn't invited | [Invite them](../../users-permissions/), or switch provisioning to **JIT** — see [Provisioning & enforcement](../provisioning-and-enforcement/) | +| **Login session expired — please try again** (`SsoStateInvalid`) | The login attempt's state/session timed out or was reused | Start the sign-in again from the login page | +| **Something went wrong signing you in — please try again** (`SsoLoginFailed`) | A generic failure, or the callback was reached with no token | Retry; if it persists, open **Technical details** and check your connection settings | + +## Password-login signal + +| Behavior | What it means | What to do | +| --- | --- | --- | +| Password login redirects to the IdP instead of signing in (`SSORequiredException`) | The member is on a verified domain, where SSO is required | This is expected — finish signing in at your IdP. The owner keeps password access as a fallback; see [enforcement](../provisioning-and-enforcement/) | + +## Setup and plan errors + +These show up while you're configuring SSO, usually as a toast or inline message. + +| Error | What it means | How to fix it | +| --- | --- | --- | +| **SSO is available on paid plans** (`PaidPlanRequiredException`) | The org isn't on a paid plan | Upgrade from **Settings → [Subscription & Billing](../../subscription-billing/)** | +| **Couldn't read the IdP metadata — check the XML** (`InvalidSamlMetadata` / `InvalidSamlConnection`) | The pasted SAML metadata couldn't be parsed | Re-copy the full metadata document from your IdP, or switch to **Enter manually** | +| **Discovery URL is invalid or not allowed (must be a public https URL)** (`InvalidOidcConnection` / `UnsafeUrl`) | The OIDC discovery URL is malformed or unreachable | Use the IdP's public `https://` `.well-known/openid-configuration` URL. A [self-hosted IdP](../providers/generic/) must be reachable over public HTTPS | +| **Enter a valid domain (e.g. acme.com)** (`InvalidDomain`) | The domain you typed isn't a valid hostname | Enter a bare domain like `acme.com`, with no scheme or path | +| **Public email domains (gmail.com, etc.) can't be used** (`PublicDomainNotAllowed`) | You tried to verify a shared public domain | Use a domain your company controls | +| **This domain is already verified by another organization** (`DomainAlreadyVerified`) | Another org claimed the domain first | Contact support if the domain is genuinely yours | +| **TXT record not found or didn't match** (`DomainVerificationFailed`) | The DNS TXT record is missing, wrong, or hasn't propagated | Re-check the **Host / Name** and **Value**, wait a few minutes, then click **Verify** again — see [Domain verification](../domain-verification/) | +| **That connection was deleted — refresh and try again** (`SsoConnectionNotFound`) | The connection was removed in another tab/session | Refresh the page and retry | +| **That domain was removed — refresh and try again** (`SsoDomainNotFound`) | The domain was removed elsewhere | Refresh the page and retry | +| **Some fields are invalid — please review and try again** (`ValidationError`) | One or more form fields failed validation | The form highlights the specific fields — correct them and resubmit | + +## Provider-specific notes + +A couple of failures trace back to one IdP's quirks: + +- **Entra ID issuer mismatch.** Using the `common` or `organizations` discovery URL makes OIDC fail with an issuer mismatch (surfaced as `OidcCallbackFailed`). Use the **tenant-specific** URL `https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration`. See [Microsoft Entra ID](../providers/entra-id/). +- **Google domain restriction.** Plain Google sign-in accepts any Google account. To restrict access to your company and route by email, use a Google Workspace account with a custom domain and verify that domain — `gmail.com` cannot be verified. See [Google](../providers/google/). + +## Related + +- [SSO overview & setup](../) +- [Verify your company domain](../domain-verification/) +- [Provisioning & enforcement](../provisioning-and-enforcement/) +- [Provider guides](../providers/) diff --git a/src/content/docs/docs/reliability/_meta.json b/src/content/docs/docs/reliability/_meta.json new file mode 100644 index 00000000..18ae0029 --- /dev/null +++ b/src/content/docs/docs/reliability/_meta.json @@ -0,0 +1,5 @@ +{ + "label": "Reliability", + "order": 19, + "collapsed": true +} diff --git a/src/content/docs/docs/reliability/burn-rate-alerts.mdx b/src/content/docs/docs/reliability/burn-rate-alerts.mdx new file mode 100644 index 00000000..f2c1fdbf --- /dev/null +++ b/src/content/docs/docs/reliability/burn-rate-alerts.mdx @@ -0,0 +1,73 @@ +--- +title: "Burn-rate Alerts" +description: "Page on burn-rate spikes before the error budget runs out — four presets cover the typical responsiveness-vs-noise tradeoffs." +sidebar: + order: 6 +--- + +A **burn-rate alert** fires when an SLO's burn rate is too high over **both** a short and a long window simultaneously. The long window suppresses brief blips; the short window keeps the alert responsive. Both are required to catch real problems without paging on every momentary spike. + +Burn-rate alerts live in two places: + +- **Per SLO**, under the SLO detail page's Burn-rate alerts section — for creating, editing, and deleting alerts. +- **Workspace-wide**, on the [Burn-rate alerts page](#workspace-wide-burn-rate-alerts) (`/<workspaceId>/slos/alerts`) — the on-call surface that lists every burn-rate alert in the workspace, firing-first. + +## Anatomy of a burn-rate alert + +| Field | What it does | +|---|---| +| **Name** | Required. How the alert shows up in the list. | +| **Short window in minutes** | Required, positive integer. | +| **Long window in minutes** | Required, must be greater than short. | +| **Burn rate threshold** | Required, must be > 1. | +| **Notification channels** | Multi-select from the workspace's notification channels — at least one is required. | +| **Enabled** | Toggle. Disabled alerts show a `Disabled` chip but keep their config. | + +## The four presets + +Four preset buttons sit above the create form. Each picks a common responsiveness-vs-noise tradeoff: + +| Preset (button label) | Short / Long | Threshold | When to use | +|---|---|---|---| +| **Critical (1m / 5m, 14.4×)** | 1 m / 5 m | 14.4× | Page immediately — for very high-priority SLOs. | +| **Fast (5m / 1h, 14.4×)** | 5 m / 1 h | 14.4× | Page now — catches sudden outages. | +| **Slow (30m / 6h, 6×)** | 30 m / 6 h | 6× | Catches gradual problems before the budget runs out. | +| **Background (2h / 24h, 3×)** | 2 h / 24 h | 3× | Daily-check level for less critical SLOs. | + +Clicking a preset fills the form fields and seeds a default name (`Critical burn`, `Fast burn`, etc.). You can rename and tweak any field before saving. + +![Burn-rate alert form with presets](./images/burn-rate-alert-form.png) + +## Create a burn-rate alert + +1. Open the SLO's detail page. +2. Scroll to **Burn-rate alerts** and click **Add burn-rate alert**. +3. Select a preset (recommended) or fill in short / long / threshold manually. +4. Name the alert. +5. Select one or more **Notification channels**. +6. Leave **Enabled** on (or off if you want to stage the alert). +7. Click **Save**. + +## Workspace-wide burn-rate alerts + +The workspace-wide page at `/<workspaceId>/slos/alerts` lists every burn-rate alert across every SLO. + +Columns: **State** / **Alert** / **SLO** / **Channels**. Each row shows its parent SLO's name and SLI kind — there's no Service column, since most SLOs aren't tied to a service. + +Default sort is **firing-first** — currently-firing alerts at the top. The **State** chip reads *Firing for X*, *Quiet*, or *Disabled*. + +Filters: + +- **Firing only** toggle. The Overview hub's e.g. *"N alerts firing"* chip deep-links here with this toggle on via `?firing=1`. + +Click any row to open the parent SLO's detail page — alerts are created and edited from there. + +## Permissions + +Workspace **admin** is required to create, edit, or delete burn-rate alerts. Everyone else can read the list. + +## Related + +- [SLO detail](../slo-detail/) — where burn-rate alerts live per SLO. +- [Notification Channels](../../platform/settings/notification-channels/) — set up the channels burn-rate alerts dispatch to. +- [What is an SLO?](../what-is-an-slo/) — burn-rate definition. diff --git a/src/content/docs/docs/reliability/create-an-slo.mdx b/src/content/docs/docs/reliability/create-an-slo.mdx new file mode 100644 index 00000000..09d02209 --- /dev/null +++ b/src/content/docs/docs/reliability/create-an-slo.mdx @@ -0,0 +1,89 @@ +--- +title: "Create an SLO" +description: "Walk through the two-step SLO wizard — pick and configure an SLI, then set target and window — with a live compliance preview." +sidebar: + order: 3 +--- + +import { Steps } from '@astrojs/starlight/components'; + +Creating an SLO is a **two-step wizard**: pick the signal you want to measure (the SLI) and configure it, then set the target and window. A live preview on the right shows the projected compliance and error budget before you save. + +Workspace **admin** is required. Non-admins don't see the Create SLO button. + +## Where to start the wizard + +Four entry points lead to the same wizard at `/<workspaceId>/slos/create`: + +- **Reliability Overview → Create SLO** (top-right). +- **Reliability Overview → Quick start → Create an SLO** card. +- **Service detail → Reliability tab → Create SLO** — deep-links with `?service_id=…`, which pre-selects the **Incident availability** kind and prefills that service in the SLI (since the service lives inside the SLI, not on the SLO). +- **Alert detail → Define an SLO** action — opens the wizard with no prefill. + +A **What is an SLO?** info tooltip sits above the steps for first-time users (create mode only), defining the core terms inline. + +## The two steps + +![SLO create wizard](./images/slo-create-wizard.png) + +### Step 1 — Define the SLI + +Pick the **family** first (a card), then the specific kind: + +- **By Count** — a ratio of good ÷ total events: APM error rate, APM latency, APM request rate, Custom metric. +- **By Monitor Uptime** — uptime from incidents or a synthetic monitor: Incident availability, Synthetic uptime. +- **By Time Slices** — a custom uptime definition over time: Time slices. + +After you pick a kind, a short "When to use this" description appears and the kind-specific form renders below. For details on each kind and its fields, see [SLI kinds](../sli-kinds/). + +Only **Incident availability** asks for a service — every other kind is workspace-level. The metric and service-name pickers are free-solo, so you can also type a value that doesn't have telemetry yet. + +When you click **Next**, KloudMate validates the SLI's configuration before advancing, so you can't carry an incomplete selector into the next step. + +### Step 2 — Target and window + +<Steps> +1. **Name** the SLO (required). + +2. Set the **target %** — `0–100`, supports decimals (e.g. `99.99`). Defaults to `99.9`. + +3. Add a **description** (optional) — shown on the SLO detail page and in reports. + +4. Pick a **default window** — a segmented toggle, defaults to `30d`. Options: `1d`, `7d`, `30d`, `Calendar month`. + +5. **Enabled** toggle defaults on. Leave it off to save the SLO without starting evaluations. + +6. Watch the **preview** rail (see below) for the projected compliance %, error budget, and a *"Would breach"* chip if applicable. + +7. Click **Create SLO**. KloudMate creates the SLO and redirects you to its detail page. +</Steps> + +## The live preview + +The right-hand rail shows a **Preview** card labelled *"Not persisted."* Whenever the SLI, target, or window settle (about 800 ms after your last change), KloudMate runs a one-off evaluation and shows: + +- The projected **compliance %**, large and color-coded, with an **On track** or **Would breach** chip. +- The would-be **error budget left**, in the SLI's unit (a duration for time-based kinds, a count for ratio kinds). +- A **trend chart** when the SLI produces a per-bucket series (today that's Time slices and Synthetic uptime); other kinds show a placeholder until the SLI is evaluated. + +If a metric or service has no data over the window, the preview surfaces a non-fatal warning so you can fix the name or filters. The preview never persists anything — it doesn't create snapshots or emit metrics. + +## Editing an SLO + +The Edit form uses the same layout as Create, with both steps open at once. Differences: + +- The **SLI kind is locked** — changing the kind would mean a different SLI, so an info banner suggests creating a new SLO instead. The rest of the SLI (including the service on an incident-based SLO) is still editable. +- The **preview still runs**, showing the effect of your *unsaved* edits before you save. +- **Save only writes the fields you changed.** + +You can reach Edit from the SLO detail page header or via the row menu on the [SLOs list](../slo-detail/). + +## Deleting an SLO + +The Delete confirmation **soft-deletes** the SLO — it disappears from the list, but its data is preserved and it stops being evaluated on the next tick. Any burn-rate alerts on the SLO are automatically disabled (their configuration is kept), so restoring the SLO later brings them back intact. + +## Related + +- [SLI kinds](../sli-kinds/) — the six SLI kinds and their fields. +- [SLO detail](../slo-detail/) — what you land on after saving. +- [Burn-rate alerts](../burn-rate-alerts/) — the next thing to set up. diff --git a/src/content/docs/docs/reliability/images/burn-rate-alert-form.png b/src/content/docs/docs/reliability/images/burn-rate-alert-form.png new file mode 100644 index 00000000..15d4dee2 Binary files /dev/null and b/src/content/docs/docs/reliability/images/burn-rate-alert-form.png differ diff --git a/src/content/docs/docs/reliability/images/incident-linked-slos.png b/src/content/docs/docs/reliability/images/incident-linked-slos.png new file mode 100644 index 00000000..a738c29c Binary files /dev/null and b/src/content/docs/docs/reliability/images/incident-linked-slos.png differ diff --git a/src/content/docs/docs/reliability/images/reliability-overview-hub.png b/src/content/docs/docs/reliability/images/reliability-overview-hub.png new file mode 100644 index 00000000..c2270d51 Binary files /dev/null and b/src/content/docs/docs/reliability/images/reliability-overview-hub.png differ diff --git a/src/content/docs/docs/reliability/images/service-reliability-tab.png b/src/content/docs/docs/reliability/images/service-reliability-tab.png new file mode 100644 index 00000000..40bd5e3a Binary files /dev/null and b/src/content/docs/docs/reliability/images/service-reliability-tab.png differ diff --git a/src/content/docs/docs/reliability/images/slo-compliance-report.png b/src/content/docs/docs/reliability/images/slo-compliance-report.png new file mode 100644 index 00000000..8b2f2fc3 Binary files /dev/null and b/src/content/docs/docs/reliability/images/slo-compliance-report.png differ diff --git a/src/content/docs/docs/reliability/images/slo-create-wizard.png b/src/content/docs/docs/reliability/images/slo-create-wizard.png new file mode 100644 index 00000000..0eaa51ff Binary files /dev/null and b/src/content/docs/docs/reliability/images/slo-create-wizard.png differ diff --git a/src/content/docs/docs/reliability/images/slo-detail-page.png b/src/content/docs/docs/reliability/images/slo-detail-page.png new file mode 100644 index 00000000..f1c78ca1 Binary files /dev/null and b/src/content/docs/docs/reliability/images/slo-detail-page.png differ diff --git a/src/content/docs/docs/reliability/index.mdx b/src/content/docs/docs/reliability/index.mdx new file mode 100644 index 00000000..d6dd2d78 --- /dev/null +++ b/src/content/docs/docs/reliability/index.mdx @@ -0,0 +1,49 @@ +--- +title: "Reliability Overview" +description: "Track service reliability with SLOs, burn-rate alerts, and a workspace-wide hub showing what needs attention right now." +sidebar: + order: 1 + label: "Overview" +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +The **Reliability** area is where you set, monitor, and respond to reliability promises about your services. Define a service-level objective (SLO), track compliance over time, and get paged before the error budget runs out. + +This page is the workspace's reliability home — the hub you land on when you click **Reliability** in the left nav. + +## What's on this page + +The Reliability Overview hub is composed of, top to bottom: + +1. **Status strip** — three clickable chips showing total SLOs, breached SLOs, and currently firing burn-rate alerts. Each chip deep-links into the matching filtered view. +2. **Burn-rate alerts firing now** — cards for every currently-firing burn-rate alert, deduplicated by parent SLO. Empty state reads e.g. *"All alerts quiet"*. Each card opens the parent [SLO detail](./slo-detail/). +3. **Breached SLOs** — top 5 breached SLOs, compact cards with compliance / target / window. A *"See all N →"* link appears when there are more than 5 breached SLOs. +4. **Quick start** — three action cards: + - **Create an SLO** → opens the two-step wizard ([Create an SLO](./create-an-slo/)). + - **Configure burn-rate alerts** → opens the workspace-wide [Burn-rate alerts](./burn-rate-alerts/) list. + - **Build a compliance report** → opens the [SLO Compliance report](./slo-compliance-report/) flow. +5. **How SLOs work** — a single concept paragraph for first-timers. + +![Reliability Overview hub](./images/reliability-overview-hub.png) + +## How SLOs work + +An **SLO** is a reliability promise written as "this service should achieve X% over a Y-day window." You pick the signal you care about (the SLI), the target percentage, and the time window. KloudMate evaluates compliance on a schedule and alerts you when the error budget runs low. + +## Next steps + +<CardGrid> + <LinkCard title="What is an SLO?" description="Plain-English definitions of SLO, SLI, error budget, and burn rate." href="./what-is-an-slo/" /> + <LinkCard title="Create an SLO" description="Walk through the two-step wizard with a live compliance preview." href="./create-an-slo/" /> + <LinkCard title="SLI kinds" description="The six SLI kinds and when to use each." href="./sli-kinds/" /> + <LinkCard title="Burn-rate alerts" description="Page on burn-rate spikes before the error budget runs out." href="./burn-rate-alerts/" /> +</CardGrid> + +## Related + +<CardGrid> + <LinkCard title="Alerts" description="Per-event alerting and routing." href="../alerts/" /> + <LinkCard title="Incident Management" description="Linked SLOs panel on incident detail." href="../incident-management/" /> + <LinkCard title="Reports" description="Add an SLO Compliance report as a recurring deliverable." href="../reports/" /> +</CardGrid> diff --git a/src/content/docs/docs/reliability/service-reliability-tab.mdx b/src/content/docs/docs/reliability/service-reliability-tab.mdx new file mode 100644 index 00000000..cea21022 --- /dev/null +++ b/src/content/docs/docs/reliability/service-reliability-tab.mdx @@ -0,0 +1,61 @@ +--- +title: "Service Reliability tab" +description: "Reliability surfaces on the service detail page, incident detail, and alert detail." +sidebar: + order: 7 +--- + +Reliability isn't only a top-level area. KloudMate surfaces SLO context where you're already working — on the service detail page, on incident detail, and on alert detail. + +## Service detail — Reliability tab + +Every IM service detail page (`/<workspaceId>/im/services/<id>`) now has tabs: + +- **Overview** (default) — the existing stat cards and integrations table. +- **Reliability** (new) — a service-scoped Create SLO button and a table of the service's **Incident availability** SLOs. Those are the only SLOs tied to a service; metric- and APM-based SLOs are workspace-level and don't appear here. Same row shape as the workspace [SLO list](../slo-detail/). The Create SLO button deep-links into the wizard with the service prefilled (it pre-selects the Incident availability kind). Empty state with explanatory copy when the service has no incident-based SLOs yet. + +Tab is URL-synced — `?tab=overview` or `?tab=reliability`. Refreshing or sharing the link preserves the active tab. + +![Service detail Reliability tab](./images/service-reliability-tab.png) + +### SLO summary chip in the service header + +When the service has at least one **Incident availability** SLO, a small chip appears in the page header next to the existing labels: + +- e.g. `N SLOs · Y% min compliance` — healthy (the lowest compliance across the service's evaluated SLOs). +- `N SLOs · M breached` — red, when at least one is breached. + +Clicking the chip jumps to the Reliability tab. The chip is **only rendered when the service has incident-based SLOs** — services without any keep the header clean. + +## Incident detail — Linked SLOs panel + +On incident detail (`/<workspaceId>/im/incidents/<id>`), a **Linked SLOs** panel appears below the standard service / integration / escalation-policy labels — but only when the affected service has at least one **incident-based-availability** SLO. + +![Incident detail Linked SLOs panel](./images/incident-linked-slos.png) + +Each linked SLO shows as a card: + +- SLO name with **Breached** / **Disabled** chips as applicable. +- Kind + target + window. +- Current compliance %. + +Clicking a card opens the SLO detail page. + +Matching is **service-only** in v1 — every incident-based SLO on the affected service appears, even if the SLO's severity / metadata filters might exclude this specific incident. Slight over-reporting is the deliberate trade-off; re-evaluating SLO logic in the browser would be fragile. + +## Alert detail — Define an SLO + +The alert detail page header now includes a **Define an SLO** outlined button (admin-only) next to the existing actions (Ask AI, Jira integration, Edit Alert). Clicking it opens the SLO create wizard at `/<workspaceId>/slos/create`. + +:::note[Placeholder image] +Screenshot pending — alert detail page header showing the Define an SLO CTA next to Edit Alert and Ask AI. +::: +![Alert detail Define an SLO CTA](../_placeholders/screenshot-placeholder.png) + +The CTA is a discovery prompt — consider defining an SLO covering the same condition as this alert. It does **not** auto-prefill the wizard because alert configurations don't map cleanly to SLI / target / window choices. + +## Related + +- [Reliability Overview](../) — workspace-wide hub. +- [SLO detail](../slo-detail/) — where the cards link to. +- [Create an SLO](../create-an-slo/) — what Define an SLO opens. diff --git a/src/content/docs/docs/reliability/sli-kinds.mdx b/src/content/docs/docs/reliability/sli-kinds.mdx new file mode 100644 index 00000000..ff68eab2 --- /dev/null +++ b/src/content/docs/docs/reliability/sli-kinds.mdx @@ -0,0 +1,153 @@ +--- +title: "SLI Kinds" +description: "The six SLI kinds KloudMate supports — grouped into By Count, By Monitor Uptime, and By Time Slices — and when to use each." +sidebar: + order: 4 +--- + +Step 1 of the SLO wizard picks the **SLI** — the specific signal you measure compliance against. KloudMate ships **six** SLI kinds, each with its own conditional form fields. This page walks through them with a "When to use this" callout and the form fields for each. + +## How the kinds are organized + +The kind picker groups the kinds into three families (the same way Datadog frames its SLO types). Pick a family card first, then the specific kind: + +| Family | Reliability measured as | Error budget | Kinds | +|---|---|---|---| +| **By Count** | A ratio of good ÷ total events. | A count of events. | APM error rate, APM latency, APM request rate, Custom metric | +| **By Monitor Uptime** | Uptime from incidents or a synthetic monitor. | A duration (downtime). | Incident availability, Synthetic uptime | +| **By Time Slices** | The share of time a metric meets a condition. | A duration. | Time slices | + +Two things changed in how SLIs are sourced, and they're worth calling out up front: + +- **Only Incident availability references a service.** Every other kind is **workspace-level** — it measures a metric or a synthetic monitor, not an Incident-Management service. The service for an incident-based SLO is part of *that SLI's* configuration, not a property of the SLO. +- **No raw-log or raw-trace SLIs.** KloudMate doesn't scan raw logs or arbitrary traces for SLIs. If you want reliability based on logs or traces, derive a metric from them in your collector and use **Custom metric** or **Time slices**. The APM kinds read pre-aggregated service span-metrics, not raw spans. + +For the metric-based kinds (Custom metric, Time slices) the metric-name and attribute pickers suggest values discovered from your workspace's OpenTelemetry metrics. The APM kinds' service-name picker suggests services emitting span metrics. All pickers are free-solo, so you can type a value that doesn't have telemetry yet — useful when you set up an SLO before the service starts emitting. + +## By Monitor Uptime + +### Incident availability + +**When to use:** *"This service should be up at least 99.9% of the time, measured by unresolved incident duration."* + +Fields: + +- **Service** — required. The Incident-Management service whose incidents count against this SLO. This is the only place an SLO chooses a service. If the service is later deleted, it still shows in the picker as *"name (deleted)"* so the SLO keeps working. +- **Severities (optional)** — multi-select, free-text (e.g. `critical`, `high`). Leave empty to count incidents of all severities. + +The SLI is *"share of the window when no matching incident was open on this service."* Time inside an enabled one-shot **maintenance window** on the service is **excluded** from the calculation rather than counted as bad. The error budget is a **duration** (seconds of downtime). + +### Synthetic uptime + +**When to use:** *"Track a synthetic monitor's uptime — the share of time its checks pass, with downtime as the error budget."* + +Fields: + +- **Synthetic monitor** — required. Pick one of the workspace's synthetic monitors. + +Synthetic uptime is a **preset**, not a separate engine — it's stored as a **Time slices** SLI over the monitor's success signal (`kloudmate_synthetic_check_success`), with one slice per check (the slice width is set to the monitor's check frequency) and a "slice is good when the check passed" condition. Slices with no run (monitor paused or not yet running) count as good, so a paused monitor doesn't burn the budget. The error budget reads as **downtime** (a duration). + +## By Count + +The By Count kinds express reliability as **good ÷ total events**, so the error budget is a count. + +### APM error rate + +**When to use:** *"Less than 1% of requests to this service should error."* (RED's Errors.) + +Fields: + +- **Service name** — required. Picker suggests services emitting span metrics. + +The SLI is *"share of a service's spans that did **not** error."* A span counts as an error when its OpenTelemetry span status is `Error`. It's computed from the service's pre-aggregated request/error span-metrics — KloudMate doesn't scan raw traces. If you need a custom definition of "error" (e.g. specific HTTP status codes), derive an error metric in your collector and use **Custom metric** instead. + +### APM latency + +**When to use:** *"At least 99% of this service's spans should complete under 200 ms."* (RED's Duration.) + +Fields: + +- **Service name** — required. Picker suggests services emitting span metrics. +- **Latency threshold (ms)** — required. + +The SLI counts spans that finished at or below the threshold as good, all spans as the denominator. It reads the service's pre-aggregated duration **histogram**, summing the buckets whose upper bound is within the threshold — so a 200 ms threshold with a 99% target means *"at least 99% of spans should be under 200 ms."* + +### APM request rate + +**When to use:** *"This service should serve at least 60 requests/minute continuously."* (RED's Rate, time-sliced.) + +Fields: + +- **Service name** — required. Picker suggests services emitting span metrics. +- **Min requests / minute** — required. The floor. +- **Bucket size (minutes, optional)** — default `1`. Controls the time-slicing granularity; larger buckets smooth spiky traffic, smaller buckets detect short stalls. + +The window is split into buckets; each bucket counts as good if its request volume is at or above the floor, bad otherwise. **Empty buckets count as below the floor (bad)** — correct for a traffic floor, but it means a sparse metric reads as catastrophic, so the preview warns when the metric covers little of the window. + +### Custom metric + +**When to use:** *"Build an SLI from arbitrary OTLP metrics — a ratio of two metric aggregations (good ÷ total)."* + +A Custom metric SLI is a ratio of two independent metric aggregations: + +- **Good events** (the numerator) — the metric + aggregation + optional filters for the events that count as a success. +- **Total events** (the denominator) — the metric + aggregation + optional filters for all eligible events. + +Both sides can use the **same metric** split by filters (e.g. a status label `success` over no filter) or **two different metrics** (e.g. `http.requests.success` over `http.requests.total`). The error budget is a count of **events**. + +The aggregation for each side adapts to the metric you pick — see [Type-aware aggregation](#type-aware-aggregation) below. + +:::note +There is no "value threshold" mode. SLO windows read pre-aggregated rollup tables (sum/count/min/max per bucket), not individual observations, so *"% of observations under X"* can't be computed as a count. For *"% of requests under a latency threshold"* use **APM latency**; for *"% of time a metric stays under a threshold"* use **Time slices**. +::: + +## By Time Slices + +### Time slices + +**When to use:** *"A custom uptime definition — the share of time a metric (or a formula across several metrics) meets a condition."* + +This is the time-based counterpart to Custom metric: instead of counting events, it splits the window into fixed slices and scores each slice good or bad. The error budget is a **duration**. + +Fields: + +- **Queries** — one or more metric queries (labelled `a`, `b`, …), each a metric + aggregation + optional filters. Use **Add query** for more. +- **Formula (optional)** — combine the queries by id, e.g. `$a / $b`. A single-query SLI evaluates as `$a`. Use **Add formula** to reveal the field. +- **Uptime condition** — *"A slice is good when the value is `<` / `≤` / `>` / `≥` <value>."* +- **Slice width** — `1 minute` or `5 minutes`. +- **No-data policy** — how slices with no measured data count: **Good** (default — a gap isn't a breach, suits low-traffic or synthetic uptime), **Bad** (a gap is a real problem, suits always-on metrics), or **Excluded** (drop the slice from the denominator). + +Each slice's (formula) value is compared against the condition; compliance is the share of good time. + +:::note +Histogram metrics aren't supported per-slice yet. For a latency-style time-slice, use a counter ratio (e.g. errors/requests) or an average via the metric's `_sum`/`_count`. The form flags a histogram metric with a clear message. +::: + +## Type-aware aggregation + +For Custom metric and Time slices, the aggregation dropdown and its default adapt to the metric's instrument type and temporality, captured automatically when you pick the metric (the same way dashboard panels and alert rules work): + +| Metric type | Aggregations offered | Default | +|---|---|---| +| **Counter (delta)** | Sum, Rate, Last | **Sum** — totals the per-interval counts over the window. | +| **Counter (cumulative)** | Increase, Rate, Last | **Increase** — last − first counts events over the window. | +| **Gauge** | Avg, Sum, Min, Max, Last, Count… | **Avg** over the window. | +| **Histogram** | P25 … P99 | **P95** (Custom metric only — histograms aren't supported per-slice). | + +A free-typed metric whose type isn't known yet is treated as a counter (the engine's default). + +## Filter rows + +The optional filter rows on Custom metric and Time slices queries have three columns: + +- **Attribute** — autocomplete that suggests the metric's label keys. +- **Operator** — `Equals` / `Not equals` / `In (any of)` / `Not in`. +- **Value** — autocomplete that suggests values for the chosen attribute. Multi-select for `In` / `Not in`, scalar for `Equals` / `Not equals`. + +**+ Add filter** appends a new row; the trash icon removes one. + +## Related + +- [Create an SLO](../create-an-slo/) — the wizard that uses these forms. +- [SLO detail](../slo-detail/) — the page you land on after creating. +- [What is an SLO?](../what-is-an-slo/) — the vocabulary these kinds build on. diff --git a/src/content/docs/docs/reliability/slo-compliance-report.mdx b/src/content/docs/docs/reliability/slo-compliance-report.mdx new file mode 100644 index 00000000..c8ff1b13 --- /dev/null +++ b/src/content/docs/docs/reliability/slo-compliance-report.mdx @@ -0,0 +1,44 @@ +--- +title: "SLO Compliance report" +description: "Schedule a workspace-wide rollup of every enabled SLO's compliance over a chosen time range." +sidebar: + order: 8 +--- + +The **SLO Compliance** report is a source on the Reports module. It produces one row per enabled SLO with its latest compliance snapshot, error budget, and burn-rate-alert state. + +The source has **no additional configuration** — it's a workspace-wide rollup of every enabled SLO. No filters, no per-SLO toggles. For a narrower view, filter the export afterward. + +## Create an SLO Compliance report + +1. Open **Settings → Reports → Create Report**. +2. In the **Report Type** picker, select **SLO Compliance**. +3. An info banner explains: e.g. *"Workspace-wide rollup: includes every enabled SLO over the selected time range. No additional filters are required."* +4. Pick a **Time Range**, **Report Format** (PDF or XLSX), **Schedule** (Fixed or Recurring), and **Recipients** — same as other report sources. See [Create Report](../../reports/create-report/) for the full walk-through. +5. Click **Save**. + +![SLO Compliance report source](./images/slo-compliance-report.png) + +## Discovery entry point + +You can also reach this flow from the [Reliability Overview hub](../) — the **Quick-start → Build a compliance report** card deep-links into the report builder with SLO Compliance pre-selected. This means users don't need to navigate to Reports separately to set this up. + +## What the report contains + +For every enabled SLO in the workspace, the export has one row with: + +- **Service** — only for **Incident availability** SLOs (resolved from the SLI); blank for the workspace-level kinds. +- **SLO** name. +- **SLI kind**. +- **Window** — the SLO's default window. +- **Target %**. +- **Compliance %** — from the SLO's latest cached snapshot for its default window. +- **Error budget** — remaining at that snapshot, in the SLI's unit (a duration or an event count). A negative value means breached. +- **Burn-rate alert** — the worst current burn-rate-alert state on the SLO (e.g. *Firing for 2.0h*, *configured*, or *—*). +- **Status** — `OK`, `BREACHED`, or `no snapshot yet` for a newly-created SLO before its first evaluation. + +## Related + +- [Reports — Create Report](../../reports/create-report/) — the surrounding report flow. +- [SLO detail](../slo-detail/) — for ad-hoc, single-SLO views. +- [Reliability Overview](../) — the deep-link entry point. diff --git a/src/content/docs/docs/reliability/slo-detail.mdx b/src/content/docs/docs/reliability/slo-detail.mdx new file mode 100644 index 00000000..9b887107 --- /dev/null +++ b/src/content/docs/docs/reliability/slo-detail.mdx @@ -0,0 +1,57 @@ +--- +title: "SLO Detail" +description: "Anatomy of the SLO detail page — window picker, snapshot, trend chart, and burn-rate alerts." +sidebar: + order: 5 +--- + +The SLO detail page is where you read compliance and manage burn-rate alerts for a single SLO. It lives at `/<workspaceId>/slos/<id>` and is reached from the [SLOs list](../), the breached cards on the [Overview hub](../), the service detail [Reliability tab](../service-reliability-tab/), or the **Linked SLOs** panel on incident detail. + +![SLO detail](./images/slo-detail-page.png) + +## Page layout + +The page is composed of, top to bottom: + +### Header + +Name, SLI kind chip, target / window summary, and a `Disabled` chip if applicable. For an **Incident availability** SLO the header also shows a **Service** link (a soft-deleted service still renders, suffixed *"(deleted)"*); every other kind is workspace-level and shows no service. Admins see **Edit** and **Delete** buttons. + +### Window picker + +A segmented toggle: `1d` / `7d` / `30d` / `Calendar month`. The SLO's **default window** is bold and tooltipped as *"Default window — refreshed hourly."* + +Picking a different window kicks off a **one-off re-evaluation** — the SLO's scheduled evaluations keep using the default. A caption next to the picker reminds you: + +e.g. > *Picking a different window checks the SLO once for that window. The default still runs on schedule.* + +### Current snapshot card + +A header strip shows *"Current snapshot"* on the left and *"Last checked X ago"* (or *"Not checked yet"*) on the right. Below it, four metric stats in a responsive grid: + +- **Compliance** — large, color-coded percentage. Helper text: *"Share of events that met the SLI. Target X%."* +- **Error budget left** — how much more you can fail this window before the SLO breaches, in the SLI's unit: a duration for the time-based kinds (incident availability, time slices, synthetic uptime), a count of events for the ratio kinds (the APM kinds and custom metric). A negative (exhausted) budget renders as e.g. `-1.4h` / `-3.3d` rather than raw seconds. +- **Default window** — e.g. `30d`. +- **Status** — `Healthy` / `Breached` / `Not evaluated` chip with a one-line helper (e.g. *"On track — within target."* or *"Below target — error budget used up."*). + +### 30-day compliance trend chart + +A line chart over the last 30 days for the currently-selected window. + +- **Dotted points** at each evaluation, so sparse data reads as a real series. +- **Dashed horizontal target line** for instant visual reference. +- **Hover tooltip** showing exact compliance % + timestamp at the cursor. +- **Smart x-axis ticks** — single-day data shows times (`14:30`); multi-day shows dates (`May 25`). +- **Y-axis anchored from below the target to 100%**, floored at 90%. Tiny drops don't appear catastrophic, and the typical "well-above-target" case still has room. + +### Burn-rate alerts section + +A list of every burn-rate alert configured for this SLO with its firing state. Admins see an **Add burn-rate alert** button and a per-alert Edit / Delete menu. + +For the full preset set and form fields, see [Burn-rate alerts](../burn-rate-alerts/). + +## Related + +- [Burn-rate alerts](../burn-rate-alerts/) — the alerting layer over an SLO. +- [Create an SLO](../create-an-slo/) — the wizard that produced this SLO. +- [SLI kinds](../sli-kinds/) — what the SLI on this SLO is measuring. diff --git a/src/content/docs/docs/reliability/what-is-an-slo.mdx b/src/content/docs/docs/reliability/what-is-an-slo.mdx new file mode 100644 index 00000000..2d4da1e6 --- /dev/null +++ b/src/content/docs/docs/reliability/what-is-an-slo.mdx @@ -0,0 +1,37 @@ +--- +title: "What is an SLO?" +description: "Plain-English definitions of SLO, SLI, target, window, compliance, error budget, burn rate, and burn-rate alert." +sidebar: + order: 2 +--- + +An SLO (Service-Level Objective) is a concrete reliability promise about a service — a measurable target and the window you measure it over. Instead of "the API should be fast," you write "99% of API requests complete under 300 ms over the last 30 days" and let the data confirm it. + +The rest of this page defines the vocabulary the Reliability area uses so the other pages read cleanly. + +## The terms + +| Term | Plain-English meaning | +|---|---| +| **SLO** (Service-level Objective) | A reliability promise written as "this service should achieve X% over a Y-day window." | +| **SLI** (Service-level Indicator) | The specific signal being measured — e.g. spans under 200 ms, seconds without an open incident, or a good/total ratio of custom metrics. | +| **Target %** | The promise — e.g. `99.9%`. | +| **Window** | The look-back period for the calculation — a rolling 1, 7, or 30 days, or the current calendar month. | +| **Compliance %** | Where the service is right now versus the target. | +| **Error budget** | How much "bad" you can afford under the target before breaching. For the time-based SLIs (incident availability, time slices, synthetic uptime) it's a duration of downtime; for the count-based SLIs (the APM kinds and custom metric) it's a count of bad events. | +| **Burn rate** | How fast you're consuming the error budget right now relative to the rate that would keep you exactly on target. `1×` means you're on pace; `14.4×` means you're consuming a 30-day budget in a few hours. | +| **Burn-rate alert** | A notification rule that fires when burn rate is too high over **both** a short and a long window simultaneously. The long window suppresses brief blips; the short window keeps the alert responsive. | + +## How they fit together + +An SLO is a target plus a window. Each evaluation produces a compliance % (the share of good events). The gap between compliance and target gets translated into an **error budget** — the count or duration of bad events the service can absorb before falling below target. Burn rate is the rate at which the budget is being consumed. A burn-rate alert pages on that rate before the budget is fully consumed. + +## Permissions + +Workspace **admin** is required to create, edit, or delete SLOs and burn-rate alerts. Everyone else can view all reliability surfaces; the create, edit, and delete actions are hidden for non-admins. + +## Related + +- [Create an SLO](../create-an-slo/) — the two-step wizard. +- [SLI kinds](../sli-kinds/) — the six SLI kinds you can pick from. +- [Burn-rate alerts](../burn-rate-alerts/) — page on dangerous burn rates. diff --git a/src/content/docs/docs/reports/alarm-report.mdx b/src/content/docs/docs/reports/alarm-report.mdx deleted file mode 100644 index 0659f0f7..00000000 --- a/src/content/docs/docs/reports/alarm-report.mdx +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Alarm Report" -description: "Generate detailed statistics about alarms, including outages, duration, and uptime over a selected time range." -sidebar: - order: 3 ---- -An Alarm Report provides detailed statistics about a single alarm or a group of alarms, helping you track their performance and state changes over a selected time range. - -![image](./images/alarm-report-1.png) - -The report displays the following details for each alarm: - -- **Name:** The name of the alarm. -- **Current State:** The state of the alarm at the time of report generation. -- **Outages:** The total number of times the alarm was triggered during the selected time range. -- **Outage Duration:** The cumulative duration the alarm remained in an alerting state within the selected time range. -- **Uptime:** The total time the alarm remained in a normal state within the selected time range. -- **MTTR (Mean Time to Resolution):** The average time taken to resolve the alarm from an alerting state back to normal. -- **SLA Target:** The SLA percentage target set by the user. -- **SLA Achieved:** The percentage of time the alarm was in a normal state during the selected time range. -- **SLA Status:** Shows Success if SLA achieved meets or exceeds the target, or Failed if it falls below. - -:::info -SLA targets for alarms can be configured on the **Alarm Configuration** page. -::: - diff --git a/src/content/docs/docs/reports/alert-report.mdx b/src/content/docs/docs/reports/alert-report.mdx new file mode 100644 index 00000000..9d7d6b0a --- /dev/null +++ b/src/content/docs/docs/reports/alert-report.mdx @@ -0,0 +1,22 @@ +--- +title: "Alert Report" +description: "Generate detailed statistics about alerts, including outages, duration, and uptime over a selected time range." +sidebar: + order: 3 +--- +An Alert Report provides detailed statistics about a single alert or a group of alerts, helping you track their performance and state changes over a selected time range. + +![Alert Report](./images/alarm-report-1.png) + +The report displays the following details for each alert: + +- **Name** — The name of the alert. +- **Current State** — The state of the alert at the time of report generation. +- **Outages** — The total number of times the alert was triggered during the selected time range. +- **Outage Duration** — The cumulative duration the alert remained in an alerting state within the selected time range. +- **Uptime** — The total time the alert remained in a normal state within the selected time range. +- **MTTR (Mean Time to Resolution)** — The average time taken to resolve the alert from an alerting state back to normal. + +:::tip +Reliability tracking has moved to SLOs. The SLA Target, SLA Achieved, and SLA Status columns are no longer part of the Alert Report — see [Reliability](../../reliability/) to set up service-level objectives, or the [SLO Compliance report](../../reliability/slo-compliance-report/) for a scheduled rollup. +::: diff --git a/src/content/docs/docs/reports/create-report.mdx b/src/content/docs/docs/reports/create-report.mdx index b60b4a74..4bc59af2 100644 --- a/src/content/docs/docs/reports/create-report.mdx +++ b/src/content/docs/docs/reports/create-report.mdx @@ -4,7 +4,7 @@ description: "Learn how to configure report sources, schedules, formats, and rec sidebar: order: 2 --- -Users can create a report from the Reports Overview page. The Create Report page is divided into four sections. +Reports are created from the Reports Overview page. The Create Report page is divided into four sections. ![image](./images/create-report-1.png) @@ -12,13 +12,13 @@ Users can create a report from the Reports Overview page. The Create Report page ![image](./images/create-report-2.png) -- **Report Type:** Select the type of report to generate — [Alarm Report](../alarm-report/), [Incident Report](../incident-report/), or [Dashboard Report](../dashboard-report/). You can also filter alarms by tags or select specific alarms using the dropdown. +- **Report Type:** Select the type of report to generate — [Alert Report](../alert-report/), [Incident Report](../incident-report/), [Dashboard Report](../dashboard-report/), or [SLO Compliance](../../reliability/slo-compliance-report/). When generating an Alert Report, you can also filter alerts by labels or select specific alerts using the dropdown. - **Time Range:** Select the duration for which the report should be generated. - **Report Format:** Choose the output format — PDF or XLSX. ### 2. Configure Schedule -Scheduling a report is optional. Disable the Active toggle if scheduling is not required. +Scheduling is optional. Disable the **Active** toggle to skip it. Users can choose between two scheduling options: **A. Fixed** @@ -47,11 +47,11 @@ Check **Send report immediately** after saving to dispatch the report instantly ### 3. Add Recipients -Select the recipient(s) from the dropdown to whom the report should be sent. The recipient list includes all email channels configured in KloudMate’s notification channels. +Select one or more recipients from the dropdown. The recipient list includes all email channels configured in KloudMate’s notification channels. ### 4. Add Report Details -- **Name:** The title of the report, which will appear on the Reports Overview page. +- **Name:** The title of the report, displayed on the Reports Overview page. - **Description:** A brief summary describing the report’s purpose. ![image](./images/create-report-5.png) diff --git a/src/content/docs/docs/reports/index.mdx b/src/content/docs/docs/reports/index.mdx index b69c3856..0d0e6e48 100644 --- a/src/content/docs/docs/reports/index.mdx +++ b/src/content/docs/docs/reports/index.mdx @@ -11,7 +11,7 @@ Each report displays the following details: - **Name:** The name assigned to the report. - **Description:** A brief summary of the report’s purpose or contents. -- **Source:** The data source used to generate the report, such as dashboards, alarms, or incidents. +- **Source:** The data source used to generate the report, such as dashboards, alerts, incidents, or SLO compliance. - **Schedule:** The configured schedule for sending the report. This field will be empty if no schedule has been set. - **Recipients:** The email address(es) the report is sent to when scheduled. - **Active:** Indicates whether the report is currently active (green) or inactive (red). diff --git a/src/content/docs/docs/synthetic/index.mdx b/src/content/docs/docs/synthetic/index.mdx index 6fda62ec..ba5133d1 100644 --- a/src/content/docs/docs/synthetic/index.mdx +++ b/src/content/docs/docs/synthetic/index.mdx @@ -18,6 +18,19 @@ Synthetic Monitoring can proactively test endpoints and services from multiple r - **Detailed Analytics** Track success rates, response times, incidents, and historical performance. - **Maintenance Windows** Schedule downtime periods to prevent false alerts during planned maintenance. +## Whitelisting KloudMate IPs + +If your endpoint or service is behind a firewall or blocks requests from public networks, allow traffic from the KloudMate synthetic IP addresses below so that checks can reach your target. These IPs are distributed across multiple regions, so whitelist all of them to ensure checks succeed from every monitoring location. + +| Region | IP Address | +| ------ | ---------------- | +| Asia | `34.14.170.159` | +| Asia | `34.93.104.208` | +| Europe | `34.140.157.247` | +| Europe | `35.195.21.112` | +| US | `34.61.24.127` | +| US | `34.42.22.136` | + ## Main Components diff --git a/src/content/docs/docs/troubleshooting.mdx b/src/content/docs/docs/troubleshooting.mdx index 6dbaa2ea..b31399b9 100644 --- a/src/content/docs/docs/troubleshooting.mdx +++ b/src/content/docs/docs/troubleshooting.mdx @@ -4,127 +4,116 @@ description: "Frequently Asked Questions about KloudMate observability, billing, sidebar: order: 20 --- + # General FAQs ## What is Observability? -In Cloud Computing, observability is about the use of tools to accumulate, correlate and analyse streams of data from a distributed application and its underlying infrastructure. This provides various [benefits](https://blog.kloudmate.com/top-5-commandments-of-observability-in-2022-7059d19cb6d2), such as effective monitoring, troubleshooting and debugging of an application, allowing you to meet service level agreements (SLAs) and business needs. +In distributed systems, observability is the practice of using tools to aggregate, correlate, and analyze streams of data from applications and the underlying infrastructure. This provides various [benefits](https://blog.kloudmate.com/top-5-commandments-of-observability-in-2022-7059d19cb6d2), such as effective monitoring, troubleshooting, and debugging of applications — helping teams meet service level agreements (SLAs) and business goals. -Observability comes packed with features such as the ability to log, monitor, & trace different servers, applications, system and data processes, across the lifecycle of the environment. +Observability relies on the ability to log, monitor, and trace different servers, applications, systems, and data processes across the lifecycle of the environment. ## Is Observability the same as Monitoring? -Monitoring only helps with regular, pre-configured system health checks, identifying problems and providing on-time, reactive notifications, to fix issues. +Monitoring helps with regular, pre-configured system health checks — identifying problems and providing reactive notifications to fix issues. -But, an ‘Observable’ system goes a step ahead and provides you with a deeper understanding into the 'Why' of any system. Observability helps explore and understand patterns that have not been pre-defined in the system. +Observability goes a step further by helping you understand *why* something went wrong. It allows you to explore and understand patterns that haven't been pre-defined, giving you deeper insight into complex, distributed systems. -An [Observability platform for Serverless Apps](https://blog.kloudmate.com/observability-and-its-role-in-overcoming-challenges-in-lambda-monitoring-ce0ef76c5581) like KloudMate, ensures you don’t just Monitor, but Observe the entire lifecycle of transactions, across several systems. All in a simple, intuitive & comprehensive interface. +KloudMate ensures you don't just monitor, but observe the entire lifecycle of transactions across your stack — all in a single, unified interface. ## What is KloudMate? -KloudMate is an all-in-one observability platform, that offers a revolutionary new way to gain deep visibility into your AWS resources, attain faster insights, or automate problem identification & real-time alerts. [Click here](../getting-started/what-is-kloudmate/) to learn more. +KloudMate is an AI-powered observability and agentic SRE-Ops platform. It helps developers identify and troubleshoot issues, optimize application performance, and gain end-to-end visibility into distributed systems and infrastructure. + +KloudMate combines AI, eBPF, and OpenTelemetry to provide a single-source-of-truth for logs, metrics, traces, and more — across cloud, on-premise, and hybrid environments. [Click here](../getting-started/what-is-kloudmate/) to learn more. ## How do I set up KloudMate? -Setting up KloudMate takes less than 5 minutes! [Click here](../aws-integration/account-setup/) to follow a step-by-step process to integrate KloudMate with your AWS environment. +Getting started with KloudMate takes just a few minutes. [Follow this guide](../getting-started/setting-up-kloudmate/) to connect your environment and start sending data to KloudMate. ## Do I need to change my code? -Integration with KloudMate requires no code changes at your end. Just [follow this link](../aws-integration/account-setup/) to see how you can integrate your AWS account(s) with KloudMate in less than 5 minutes! +Not necessarily. KloudMate supports auto-instrumentation via the KloudMate Agent and eBPF, which requires no code changes. For deeper instrumentation, OpenTelemetry-based manual instrumentation is also available for most languages and frameworks. ## How is my data stored in KloudMate? -KloudMate works on top of AWS CloudWatch to collect logs and metrics. All the imported data is then encrypted and stored in a dedicated and secure database, hosted on AWS. +All data ingested into KloudMate is encrypted and stored in a dedicated, secure database hosted on AWS. Data is encrypted both at-rest and in-transit. ## How can I be sure that my data is secure with KloudMate? -As users of our own product, we understand how important the security and privacy of your data is. - -While all data is encrypted both at-rest and in-transit, we also track all the latest security best practices around the globe, promptly upgrade our services to fix new vulnerabilities and constantly make sure we are using the latest technologies available. - -We work closely with industry leaders in web application and infrastructure security domains, who perform penetration tests and audits of [kloudmate.com](http://kloudmate.com/) +KloudMate follows the latest security best practices, promptly upgrades services to address new vulnerabilities, and works with industry leaders in web application and infrastructure security who conduct regular penetration tests and audits of [kloudmate.com](https://www.kloudmate.com). You can review KloudMate's security posture at [kloudmate.com/security](https://www.kloudmate.com/security). ## How much does KloudMate cost? -We are currently in the Beta phase, inviting select customers to join the platform and share their experience with us. - -Customers who have early access to KloudMate will be able to use it for free during the beta, and receive 6 months worth of usage credits during commercial launch, thereafter. [Click here](https://app.kloudmate.com/signup) to go to the sign up page. +KloudMate offers a free plan as well as paid subscription tiers. Visit [kloudmate.com/pricing](https://www.kloudmate.com/pricing) for current plan details and pricing. -Our pricing will remain simple and upfront, with subscription plans that suit all kinds of usage patterns. +## What does KloudMate monitor? -## Which AWS Serverless services does KloudMate support? +KloudMate supports observability across a wide range of environments and workloads, including: -KloudMate currently integrates with over 15 most widely used serverless services, such as AWS Lambda, API Gateway, DynamoB, Kinesis Stream, S3, ECS, EKS, SQS, Firehose and more! +Infrastructure (hosts, VMs, Kubernetes, Docker) +Cloud services (AWS, Azure, and more) +Databases (PostgreSQL, MySQL, MongoDB, Redis, OracleDB, MSSQL) +Applications and microservices (via OpenTelemetry and auto-instrumentation) +Serverless functions (AWS Lambda and others) +Real User Monitoring (RUM) +Synthetic Monitoring +LLM Observability +Network Monitoring -## I don't use Serverless workloads. Is KloudMate for me? - -KloudMate is an all-in-one Monitoring and Observability tool. Serverless functionalities aside, KloudMate can be used to manage and monitor your cross-region cloud inventory, performance, cost and billing insights, and much more. +## What programming languages does KloudMate support? -## Is there any overhead or latency added to my Lambdas? +KloudMate supports instrumentation for most major languages via OpenTelemetry, including Node.js, Python, Java, Go, .NET, PHP, Ruby, and React. Auto-instrumentation via eBPF is also available for zero-code observability. -KloudMate does not affect the performance of your lambda functions in any way. It only requires read access and collects all the information from AWS CloudWatch. +## Is there any overhead added to my applications? -## What programming languages does KloudMate support? +KloudMate's eBPF-based observability is non-intrusive and does not affect the performance of your applications. The KloudMate Agent is lightweight and designed for minimal resource overhead. -KloudMate supports most of the programming languages that are supported by AWS Lambda, including Node.js, Python, JAVA, C#, Ruby, among others. +## Do team members need access to the cloud console? -## Do employees using KloudMate also need access to the AWS console? +No. KloudMate provides a unified interface for monitoring your entire environment, regardless of whether individual team members have direct access to your cloud provider's console. -The beauty of KloudMate is that you can access crucial information about your Cloud environment, and monitor all your AWS resources across several regions using KloudMate's AWS inventory even if you don't have access to the AWS console. +## How does KloudMate help engineering teams? -## How does KloudMate help businesses using AWS? - -KloudMate can prove to be a business critical application for those using the AWS Cloud to deploy their infrastructure and applications. With the help of KloudMate, businesses can improve performance and cost strategies, monitor cloud resources and debug applications hosted on serverless infrastructure, to name a few. +KloudMate helps teams detect and resolve issues faster, reduce observability costs (by over 60% compared to traditional APM tools), automate diagnosis and triage, and collaborate more effectively — all from a single platform. ## How do I get technical support from KloudMate? -You can click on the Chat icon on the website (also available within the application), to talk to us in real-time, or [drop us an email](mailto:hello@kloudmate.com) and someone will get back to you in no time. +You can use the chat icon within the application or website to reach the team in real-time, or [send an email](mailto:hello@kloudmate.com). You can also join the [o11y Lounge Slack community](https://join.slack.com/t/o11y-lounge/shared_invite/zt-zvzq985i-Khwnq2jsm5uLTdBe87gCgw) to connect with the KloudMate team and other users. -Looking for more? Join our [Slack Community](https://kloudmate.slack.com/join/shared_invite/zt-zvzq985i-Khwnq2jsm5uLTdBe87gCgw#/shared-invite/email) to share your experience and interact with the core team members, fellow developers and customers. +--- # Billing FAQs -## Is there a free plan? - -Yes! We want everyone to experience what Observability can do for them. And so, we have a generous free tier that you can instantly [SignUp](https://app.kloudmate.com/signup) for. No credit card needed, no questions asked. Free, for life! - -## Is the free plan, really free? - -The free plan is absolutely free! It offers 1GB of log ingestion, 1 million Lambda invocations, and 3 days of data retention. For most Developers exploring Serverless to deploy applications, this should be good enough, even for the long term. - -## What are the various 'Usage' factors that a plan comprises of? - -There are 3 types of usage associated with all the plans: - -- Number of Lambda executions (measured in millions) -- Log Analytics (measured in GB) -- Log retention (measured in days) - -## What does KloudMate consider as an 'Execution'? - -An execution is when Lambda invokes a function successfully, by providing a secure, isolated runtime environment. The environment manages the resources that are required to run your function and complete the task. - -## What happens when I reach my usage limits? - -If you are on a free account, your usage will be automatically paused (unless upgraded to a higher plan). The limits of the free plan reset every billing cycle, allowing you to resume and continue using the platform. +## Do I need to provide a credit card to sign up for a free trial? +No. You can start on the Free plan or begin a 14-day Pro trial without a credit card. Billing details are only needed when you decide to continue on a paid plan. -Our paid plans on the other hand, have an 'overage' component, that you can pay for as you go. +## How do I sign up for KloudMate? +Create an account at app.kloudmate.com/signup, then send data using OpenTelemetry, CloudWatch, or one of our integrations. You will start seeing your logs, metrics, and traces within minutes. -## Can I invite others to join my account? +## How does the billing work? +KloudMate uses purely usage-based pricing. You are billed monthly for the volume of logs and traces you ingest (per GB) and metrics (per million samples), plus any extended data-retention you choose. The Pro plan starts at $99 / month. -Yes! The powers of KloudMate's observability tools are best used with multiple eyes looking at the intelligence it offers. Every KloudMate plan allows you to invite unlimited users, colleagues or team members, so you can work together and make informed, collaborative decisions. +## Is pricing per user or per host? +No. Every plan includes unlimited users, with no per-host or per-agent fees. You pay only for the telemetry you send — logs and traces by volume, and metrics by sample count. -## Our usage of Serverless is fairly high. Can we get a customized plan? +## How does data-retention pricing work? +Each plan includes a default retention window, and you can extend it. Longer retention for logs and traces raises their storage cost by the multiplier shown in the calculator, so you can trade off how long you keep data against cost. Metrics are billed by sample count, so their retention does not change the price. -Absolutely! Feel free to reach out to our [Customer Success team](mailto:cs@kloudmate.com?subject=Customized%20Plan%20Enquiry) and they'll design something tailored for your usage, in no time! +## How do AI credits work? +KloudMate Assistant runs on credits. A chat message costs 1 credit, and a full investigation costs 2. Every plan includes a monthly allowance — 5 credits on Free and 20 on Pro — and you can buy prepaid credits any time you need more. Enterprise teams can bring their own AI provider key instead of using credits. -## There are only monthly and annual subscription plans. We'd like the option to pay quarterly in advance -We understand certain Enterprises prefer to pay quarterly in advance, for various reasons. Please [reach out to us](mailto:enterprise@kloudmate.com?subject=Customized%20Plan%20Enquiry) and we'll have it taken care of. +## Do you offer annual billing? +Yes. Annual billing is available at a discount compared with paying monthly. Get in touch and we will set it up for you. -## Is there added cost to my AWS bill? +## What payment methods are accepted? +We accept all major credit and debit cards. Enterprise customers can also pay by invoice — talk to us and we will set that up. -KloudMate uses AWS APIs to gather data from your cloud components. For AWS Lambda function Logs, AWS charges data transferred out of CloudWatch equivalently to data transferred out of an EC2 server. Check it out at AWS CloudWatch pricing. The total cost should represent a small percentage of your total AWS bill. +## Can I change plans at any time? +Yes. You can upgrade, downgrade, or adjust retention whenever you need, and the change is reflected in your usage-based bill. There are no long-term lock-ins on the Free or Pro plans. -For Alarms, we use CloudWatch GetMetricData API to fetch metrics to evalulate alarm. You'll be billed based on how many alarms you set and how frequently you've configured to evaluate the alarm. +## Can we run KloudMate in our own cloud? +Yes. KloudMate Infinity runs the full platform inside your own cloud account, deployed and operated by KloudMate, so your telemetry never leaves your network. It is available on Enterprise plans. -For other cloud components (SQS, ECS, DynamoDB, etc), the cost will depend on how frequently you determine the polling frequency. Each API request will be billed according to CloudWatch API pricing set by AWS. +## Our usage is fairly high. Can we get a customized plan? +Absolutely. For high-volume workloads we offer custom pricing, custom retention, dedicated support, and bring-your-own-key AI on the Enterprise plan. Get in touch and we will tailor a plan to your needs. diff --git a/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alarm-list-panel.mdx b/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alarm-list-panel.mdx deleted file mode 100644 index 0d97371d..00000000 --- a/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alarm-list-panel.mdx +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: "Alarm List Panel" -description: "Learn how to configure and use this visualization feature in KloudMate." ---- -The Alarms List Panel displays a list of configured alarms within the selected time range. The list can be filtered based on available attributes to focus on specific alarms relevant to your dashboard. - -When the panel type is set to Alarms List, the following settings are available under the **General** tab: - -![image](./images/alarm-list-panel-1.jpeg) - -### Panel - -- **Name:** Enter a name for your panel. This appears as the panel title on the dashboard. -- **Description:** Enter a description to provide additional context about what the panel displays. - -### Time Range - -Configure time range behavior for this panel independently from the dashboard. - -- **Override Dashboard Time Range:** Individual panels can use their own time range configuration. Enable this to override the time range set at the dashboard level. -- **Time Shift:** Shift the panel’s start and end time by a specified time shift expression. Use the minus (`-`) operator to subtract time, with the same units supported by the dashboard time range expressions. - -Refer to [Time Range Expressions and Settings](../time-range-expressions-and-settings/) for supported units and syntax. diff --git a/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alert-list-panel.mdx b/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alert-list-panel.mdx new file mode 100644 index 00000000..65a39b87 --- /dev/null +++ b/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/alert-list-panel.mdx @@ -0,0 +1,23 @@ +--- +title: "Alert List Panel" +description: "Configure and use the Alert List panel in KloudMate dashboards." +--- +The Alert List Panel displays a list of configured alert rules within the selected time range. The list can be filtered based on available attributes to focus on specific alerts relevant to your dashboard. + +When the panel type is set to Alerts List, the following settings are available under the **General** tab: + +![Alert List panel settings](./images/alarm-list-panel-1.jpeg) + +### Panel + +- **Name:** Enter a name for your panel. This appears as the panel title on the dashboard. +- **Description:** Enter a description to provide additional context about what the panel displays. + +### Time Range + +Configure time range behavior for this panel independently from the dashboard. + +- **Override Dashboard Time Range:** Individual panels can use their own time range configuration. Enable this to override the time range set at the dashboard level. +- **Time Shift:** Shift the panel's start and end time by a specified time shift expression. Use the minus (`-`) operator to subtract time, with the same units supported by the dashboard time range expressions. + +Refer to [Time Range Expressions and Settings](../time-range-expressions-and-settings/) for supported units and syntax. diff --git a/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/index.mdx b/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/index.mdx index 2d396c23..ebaa7a11 100644 --- a/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/index.mdx +++ b/src/content/docs/docs/visualize-data/dashboards/adding-a-panel/index.mdx @@ -35,7 +35,7 @@ From your dashboard, click **Add Panel**. A dialog appears with two options: ![image](./images/adding-a-panel-4.jpeg) -For more information on setting up queries, see [Setting Up KloudMate Alarms](../../../alarms/create-alarms/). For expressions, see [Writing Expressions for KloudMate Alarms](../../../alarms/expressions/). +For more information on setting up queries, see [Creating Alerts](../../../alerts/create-alerts/). For expressions, see [Alert Expressions](../../../alerts/expressions/). 3. Click **Run Query** to execute the query and preview the result in the visualization area. Running the query lets you verify the data looks correct before saving the panel. @@ -45,7 +45,7 @@ For more information on setting up queries, see [Setting Up KloudMate Alarms](.. Choosing the right type makes data easier to interpret, for example, use TimeSeries for trends over time or Stats for a single current value. -*Available types are:* **TimeSeries** , **Stats** , **Table** , **Pie Chart** , **Bar Chart** , **Histogram** , **Logs List** , **Spans List** , **Alarms List** , **Issues List** , and **Text**. +*Available types are:* **TimeSeries** , **Stats** , **Table** , **Pie Chart** , **Bar Chart** , **Histogram** , **Logs List** , **Spans List** , **Alerts List** , **Issues List** , and **Text**. ![image](./images/adding-a-panel-5.jpeg) diff --git a/src/content/docs/docs/visualize-data/dashboards/dashboard-details.mdx b/src/content/docs/docs/visualize-data/dashboards/dashboard-details.mdx index 7e6a34f2..e87c3e8b 100644 --- a/src/content/docs/docs/visualize-data/dashboards/dashboard-details.mdx +++ b/src/content/docs/docs/visualize-data/dashboards/dashboard-details.mdx @@ -5,7 +5,7 @@ description: "Understand the dashboard details view, toolbar controls, and shari sidebar: order: 3 --- -Opening a dashboard takes you to the Dashboard Details view where all your panels are displayed together. This is your live monitoring view, all panels update based on the selected time range and refresh settings. +Opening a dashboard takes you to the Dashboard Details view where all your panels are displayed together. This is your live monitoring view — all panels update based on the selected time range and refresh settings. ![image](./images/dashboard-details-1.png) @@ -31,13 +31,13 @@ The toolbar at the top provides the following controls: - **Fullscreen:** Expands the dashboard to fullscreen view. - **Edit:** Enters edit mode to modify the dashboard and its panels. -## Viewing Alarms in Dashboards +## Viewing Alerts in Dashboards -KloudMate integrates alarms directly into your dashboard visualizations, providing immediate context on system health alongside your metrics. +KloudMate integrates alerts directly into your dashboard visualizations, providing immediate context on system health alongside your metrics. -- **Panel Alarm Indicators:** If a panel is linked to an alarm and the alarm is currently **Firing**, a status indicator will appear on the panel header. -- **Alarms Drawer:** Clicking on an alarm indicator opens the Alarms Drawer without leaving the dashboard context. This drawer displays information for alarms in the **Firing** state: - - The current evaluation state of the alarm. +- **Panel Alert Indicators:** If a panel is linked to an alert and the alert is currently **Firing**, a status indicator will appear on the panel header. +- **Alerts Drawer:** Clicking on an alert indicator opens the Alerts Drawer without leaving the dashboard context. This drawer displays information for alerts in the **Firing** state: + - The current evaluation state of the alert. - The conditions that triggered it. ## Sharing a Dashboard @@ -58,8 +58,8 @@ You can replicate a dashboard in another workspace by exporting it as JSON. This To share: -1. Open the dashboard and click the more options icon, then select **Share**. This copies the dashboard as a JSON to your clipboard. +1. Open the dashboard and click the more options icon, then select **Share**. This copies the dashboard JSON to your clipboard. ![image](./images/dashboard-details-4.jpeg) -2. Navigate to the Dashboards section in the target workspace, click **Import** , paste the copied JSON, and click **Submit**. The dashboard will be recreated as a replica in the new workspace. +2. Navigate to the Dashboards section in the target workspace, click **Import** , paste the copied JSON, and click **Submit**. The dashboard is recreated in the new workspace. diff --git a/src/content/docs/docs/visualize-data/explore/index.mdx b/src/content/docs/docs/visualize-data/explore/index.mdx index 132e7547..34b060b9 100644 --- a/src/content/docs/docs/visualize-data/explore/index.mdx +++ b/src/content/docs/docs/visualize-data/explore/index.mdx @@ -3,7 +3,7 @@ title: "Explore" description: "Query, aggregate, and visualize metrics in KloudMate Explore." sidebar: label: "Overview" - order: 1 + order: 0 --- import { Image } from 'astro:assets'; @@ -17,7 +17,7 @@ import exploreQueries from './images/explore-queries.png'; import exploreRun from './images/explore-run.png'; import exploreSave from './images/explore-save.png'; -Explore is the ad hoc query workspace in KloudMate. It lets you build and validate queries before saving them into a dashboard, with support for metrics, logs, and traces. +Explore is the ad hoc query workspace in KloudMate. It lets you build and validate queries before saving them to a dashboard, with support for metrics, logs, and traces. This walkthrough focuses on metrics. @@ -65,7 +65,7 @@ This walkthrough focuses on metrics. 5. **Add Queries and Expressions** Use **Add Query** to compare multiple metric series in one panel. - Use **Add Expression** to derive values from existing queries, including math, reduce, and condition expressions. See [Writing Expressions for KloudMate Alarms](../../alarms/expressions/) for the full expression syntax. + Use **Add Expression** to derive values from existing queries, including math, reduce, and condition expressions. See [Alert Expressions](../../alerts/expressions/) for the full expression syntax. <Image src={exploreQueries} alt="Queries and Expressions" /> diff --git a/src/content/docs/docs/visualize-data/index.mdx b/src/content/docs/docs/visualize-data/index.mdx index 76b7fffa..27e17167 100644 --- a/src/content/docs/docs/visualize-data/index.mdx +++ b/src/content/docs/docs/visualize-data/index.mdx @@ -3,7 +3,7 @@ title: "Visualize Data" description: "Create custom dashboards and explore telemetry data using KloudMate." sidebar: label: "Overview" - order: 1 + order: 0 --- KloudMate provides powerful visualization tools to help you monitor, analyze, and extract insights from your telemetry data. Whether you need a persistent view of critical infrastructure KPIs or an ad-hoc workspace to troubleshoot an ongoing incident, KloudMate's visualization suite has you covered. diff --git a/src/content/docs/guides/index.mdx b/src/content/docs/guides/index.mdx index ba86fc90..2226792d 100644 --- a/src/content/docs/guides/index.mdx +++ b/src/content/docs/guides/index.mdx @@ -13,6 +13,12 @@ Walkthroughs for sending application logs directly to KloudMate. - [Send Winston Logs with OpenTelemetry (Node.js)](/guides/logs/send-winston-logs-with-opentelemetry-nodejs/) +## LLM & AI Guides + +Instrument LLM and agent applications and send their traces to KloudMate. + +- [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/) + ## OpenTelemetry Instrumentation Tutorials Step-by-step tutorials on how to instrument sample applications from scratch using OpenTelemetry zero-code and SDK instrumentation. diff --git a/src/content/docs/guides/llm/_meta.json b/src/content/docs/guides/llm/_meta.json new file mode 100644 index 00000000..4aad6589 --- /dev/null +++ b/src/content/docs/guides/llm/_meta.json @@ -0,0 +1,4 @@ +{ + "label": "LLM & AI", + "order": 3 +} diff --git a/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx b/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx new file mode 100644 index 00000000..a72cad6c --- /dev/null +++ b/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx @@ -0,0 +1,172 @@ +--- +title: "Instrument a Python LLM App with OpenLLMetry" +description: "Trace a LangChain agent end to end with OpenLLMetry and send prompts, tool calls, token usage, and latency to KloudMate." +sidebar: + order: 1 +--- + +This guide walks you through instrumenting a real LLM application — a small LangChain agent that calls tools — and viewing its traces in KloudMate. By the end you'll see each agent run as a single trace: the model's reasoning, every tool call, token usage per call, and where the time went. + +You'll build a customer-support agent for an online store. The agent answers a question by deciding which tools to call (look up an order, check a return policy), then writing a reply. That back-and-forth is exactly the kind of multi-step flow that's hard to debug from logs alone — and easy to read as a trace. + +OpenLLMetry does the instrumentation. It's an OpenTelemetry-native SDK from Traceloop that auto-instruments LangChain and the underlying model calls, so you add a few lines of setup and change nothing in the agent itself. For the concepts behind it, see [Introduction to OpenLLMetry](/llm-observability/what-is-openllmetry/). + +## Prerequisites + +- Python 3.9 or later. +- An OpenAI API key ([where to find it](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key)). +- A KloudMate workspace API key, from **Settings → API Keys**. + +## Step 1: Set up the project + +Create a directory, activate a virtual environment, and install the packages: + +```bash +mkdir support-agent && cd support-agent +python3 -m venv venv +source ./venv/bin/activate +pip install traceloop-sdk langchain langchain-openai +``` + +`traceloop-sdk` brings in OpenLLMetry and the OpenTelemetry exporter. The LangChain packages are the app itself. + +## Step 2: Point OpenLLMetry at KloudMate + +OpenLLMetry exports over OTLP/HTTP, so it needs your KloudMate endpoint and API key. Set them as environment variables so you don't hard-code secrets: + +```bash +export OPENAI_API_KEY="YOUR_OPENAI_API_KEY" +export KM_API_KEY="YOUR_KLOUDMATE_API_KEY" +``` + +You'll initialize OpenLLMetry once, before the agent runs. A single `Traceloop.init()` call wires up the exporter and instruments LangChain and OpenAI automatically: + +```python +import os +from traceloop.sdk import Traceloop + +Traceloop.init( + app_name="support-agent", + api_endpoint="https://otel.kloudmate.com:4318", # the SDK appends /v1/traces + headers={"Authorization": os.environ["KM_API_KEY"]}, + disable_batch=True, # export each span as it finishes while you're testing +) +``` + +:::note +KloudMate authenticates with your API key in the `Authorization` header — the raw key, with no `Bearer` prefix. Keep `disable_batch=True` for local testing so traces show up immediately; remove it in production to batch spans and export them in the background. +::: + +## Step 3: Build the agent + +Create `app.py`. The agent has two tools backed by in-memory data so the example runs without a database. In a real app these would be API or database calls — but the tool *shape* is what matters, because that's what shows up in the trace. + +```python +import os +from traceloop.sdk import Traceloop + +# Initialize OpenLLMetry BEFORE importing/using LangChain so the calls are traced. +Traceloop.init( + app_name="support-agent", + api_endpoint="https://otel.kloudmate.com:4318", + headers={"Authorization": os.environ["KM_API_KEY"]}, + disable_batch=True, +) + +from langchain.agents import AgentExecutor, create_tool_calling_agent +from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder +from langchain_core.tools import tool +from langchain_openai import ChatOpenAI + +# --- Mock data (stand-ins for real APIs) --- +ORDERS = { + "4412": {"status": "Delivered", "category": "audio", "total": 99.95}, +} +RETURN_POLICIES = { + "audio": {"window_days": 30, "restocking_fee_pct": 10}, + "default": {"window_days": 30, "restocking_fee_pct": 0}, +} + +# --- Tools --- +@tool +def get_order_status(order_id: str) -> dict: + """Look up an order's status, category, and total by its order ID.""" + order = ORDERS.get(order_id.strip().lstrip("#")) + return order or {"error": f"No order found with id {order_id}."} + +@tool +def get_return_policy(category: str) -> dict: + """Get the return window and restocking fee for a product category.""" + return RETURN_POLICIES.get(category.lower(), RETURN_POLICIES["default"]) + +TOOLS = [get_order_status, get_return_policy] + +# --- Agent --- +SYSTEM_PROMPT = ( + "You are ShopMate, a concise customer-support agent. " + "Use the tools to look up real order and policy data — never guess order " + "details or refund amounts. Give the customer a short, helpful answer." +) + +def build_agent() -> AgentExecutor: + llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) + prompt = ChatPromptTemplate.from_messages( + [ + ("system", SYSTEM_PROMPT), + ("human", "{input}"), + MessagesPlaceholder("agent_scratchpad"), + ] + ) + agent = create_tool_calling_agent(llm, TOOLS, prompt) + return AgentExecutor(agent=agent, tools=TOOLS, verbose=False) + +if __name__ == "__main__": + agent = build_agent() + result = agent.invoke( + {"input": "I'd like to return order #4412. What's the policy and how much would I get back?"} + ) + print(result["output"]) +``` + +## Step 4: Run it + +```bash +python3 app.py +``` + +The agent answers the question, and OpenLLMetry exports the trace to KloudMate. Behind that one answer, the agent made two model calls and two tool calls — all captured as a single trace. + +## Step 5: View the trace in KloudMate + +Open [KloudMate](https://app.kloudmate.com), go to **Traces**, and filter by the service `support-agent`. Open the most recent trace. + +You'll see the full agent run as a waterfall: the `AgentExecutor` at the top, the model calls (`ChatOpenAI.chat`), and the tool calls (`get_order_status`, `get_return_policy`) nested underneath with their durations. Select any model span to read the exact prompt and response, the token usage, and the finish reason. + +For a tour of everything KloudMate surfaces on an AI trace — the conversation transcript, token and cache breakdown, tool calls, and the AI Flow graph — see [AI Trace Observability](/llm-observability/ai-trace-observability/). + +## Step 6: Tag traces with a user and session + +Production agents serve many customers across many conversations. Attach a user and session to every span so you can find one customer's traces later, or follow a single conversation end to end. Set these association properties before each run: + +```python +Traceloop.set_association_properties( + {"user_id": "cust_1042", "session_id": "sess_8f21"} +) +result = agent.invoke({"input": "..."}) +``` + +KloudMate stores these on every span in the run, so you can filter and group traces by `user_id` or `session_id` when you investigate an issue. + +## Step 7: Prepare for production + +A few changes once you move past local testing: + +- **Batch exports.** Drop `disable_batch=True` so spans export in the background instead of one HTTP request per span. +- **Decide what to capture.** OpenLLMetry records prompt and completion content by default, which is what makes the conversation view useful. If your prompts carry sensitive data, set `TRACELOOP_TRACE_CONTENT=false` to record metadata (tokens, model, latency) without the message bodies. +- **Keep init first.** Call `Traceloop.init()` before the rest of your app imports run, so every model and tool call is instrumented. + +## Related resources + +- [AI Trace Observability](/llm-observability/ai-trace-observability/) — what to look for once your traces land in KloudMate +- [Introduction to OpenLLMetry](/llm-observability/what-is-openllmetry/) +- [Instrument a Python App](/guides/opentelemetry/python/) — general OpenTelemetry instrumentation diff --git a/src/content/docs/index.mdx b/src/content/docs/index.mdx index 3276bb4e..24c8e852 100644 --- a/src/content/docs/index.mdx +++ b/src/content/docs/index.mdx @@ -74,9 +74,9 @@ KloudMate is an end-to-end observability platform that gives your team unified v href="visualize-data/" /> <LinkCard - title="Alarms & Notifications" - description="Set up intelligent alerts, liquid templating, and ChatOps routing." - href="alarms/" + title="Alerts & Notifications" + description="Create alert rules, group related signals, and route notifications through KloudMate Incidents, Slack, and more." + href="alerts/" /> </CardGrid> @@ -84,9 +84,14 @@ KloudMate is an end-to-end observability platform that gives your team unified v <CardGrid> <LinkCard title="Incident Management" - description="Manage issue lifecycles, services, and coordinate team responses." + description="Manage incident lifecycles, services, and coordinate team responses." href="incident-management/" /> + <LinkCard + title="Reliability & SLOs" + description="Track service reliability with SLOs, burn-rate alerts, and compliance reports." + href="reliability/" + /> <LinkCard title="Guides & Instrumentations" description="Step-by-step OpenTelemetry instrumentation tutorials for your code."