diff --git a/.github/workflows/image-transfer-dev2.yml b/.github/workflows/image-transfer-dev2.yml new file mode 100644 index 00000000..906f86e6 --- /dev/null +++ b/.github/workflows/image-transfer-dev2.yml @@ -0,0 +1,49 @@ +name: Transfer images to mosipdev2 + +# Stage-scoped transfer: destination is fixed to mosipdev2. +# Requires GitHub Environment "transfer-dev2" with required reviewers +# and Environment secret MOSIPDEV2_DOCKER_TOKEN. +# See: release/docs/github-environments-image-transfer.md + +on: + workflow_dispatch: + inputs: + USERNAME: + description: 'Registry username (Docker Hub user or Harbor robot)' + required: true + default: '' + type: string + REGISTRY_URL: + description: 'Destination registry URL' + required: true + default: 'https://index.docker.io/v1/' + type: string + REGISTRY_TYPE: + description: 'Destination registry type' + required: true + default: 'dockerhub' + type: choice + options: + - dockerhub + - harbor + - other + ENABLE_WIREGUARD: + description: 'Enable WireGuard VPN (required for private Harbor networks)' + required: false + default: false + type: boolean + +jobs: + Image-transfer: + environment: transfer-dev2 + uses: mosip/kattu/.github/workflows/image-transfer.yml@master + with: + DESTINATION_ORGANIZATION: mosipdev2 + REGISTRY_URL: ${{ inputs.REGISTRY_URL }} + REGISTRY_TYPE: ${{ inputs.REGISTRY_TYPE }} + ENABLE_WIREGUARD: ${{ inputs.ENABLE_WIREGUARD }} + USERNAME: ${{ inputs.USERNAME }} + secrets: + TOKEN: ${{ secrets.MOSIPDEV2_DOCKER_TOKEN }} + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_DEVOPS }} + WIREGUARD_CONFIG: ${{ secrets.WIREGUARD_CONFIG }} diff --git a/.github/workflows/image-transfer-qa.yml b/.github/workflows/image-transfer-qa.yml new file mode 100644 index 00000000..5e64dd8b --- /dev/null +++ b/.github/workflows/image-transfer-qa.yml @@ -0,0 +1,49 @@ +name: Transfer images to mosipqa + +# Stage-scoped transfer: destination is fixed to mosipqa. +# Requires GitHub Environment "transfer-qa" with required reviewers +# and Environment secret MOSIPQA_DOCKER_TOKEN. +# See: release/docs/github-environments-image-transfer.md + +on: + workflow_dispatch: + inputs: + USERNAME: + description: 'Registry username (Docker Hub user or Harbor robot)' + required: true + default: '' + type: string + REGISTRY_URL: + description: 'Destination registry URL' + required: true + default: 'https://index.docker.io/v1/' + type: string + REGISTRY_TYPE: + description: 'Destination registry type' + required: true + default: 'dockerhub' + type: choice + options: + - dockerhub + - harbor + - other + ENABLE_WIREGUARD: + description: 'Enable WireGuard VPN (required for private Harbor networks)' + required: false + default: false + type: boolean + +jobs: + Image-transfer: + environment: transfer-qa + uses: mosip/kattu/.github/workflows/image-transfer.yml@master + with: + DESTINATION_ORGANIZATION: mosipqa + REGISTRY_URL: ${{ inputs.REGISTRY_URL }} + REGISTRY_TYPE: ${{ inputs.REGISTRY_TYPE }} + ENABLE_WIREGUARD: ${{ inputs.ENABLE_WIREGUARD }} + USERNAME: ${{ inputs.USERNAME }} + secrets: + TOKEN: ${{ secrets.MOSIPQA_DOCKER_TOKEN }} + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_DEVOPS }} + WIREGUARD_CONFIG: ${{ secrets.WIREGUARD_CONFIG }} diff --git a/release/docs/github-environments-image-transfer.md b/release/docs/github-environments-image-transfer.md new file mode 100644 index 00000000..5886d534 --- /dev/null +++ b/release/docs/github-environments-image-transfer.md @@ -0,0 +1,372 @@ +# GitHub Environments for Image Transfer — What They Are & How to Implement + +## What is a GitHub Environment? + +A **GitHub Environment** is a named deployment target in a repository (for example `transfer-dev2`, `transfer-qa`, `transfer-prod`). It is **not** a server and **not** Docker Hub. It is a **control gate** inside GitHub Actions. + +When a workflow job declares: + +```yaml +jobs: + transfer: + environment: transfer-dev2 +``` + +GitHub does three important things **before that job can run or read environment secrets**: + +1. Applies **protection rules** (required reviewers, wait timer, allowed branches). +2. Creates a **deployment record** (who triggered, who approved, when, which commit). +3. Unlocks **Environment secrets** only for that job (and only after rules pass). + +### Why this matters for image transfer + +Today, anyone who can run the manual image-transfer workflow can aim it at any org (if they pick the right secret). With Environments: + +| Without Environments | With Environments | +|---|---| +| Click **Run workflow** → job starts immediately | Click **Run workflow** → job waits for Approver | +| Repo secrets readable by any write-access workflow | Docker push token locked in Environment; released only after approval | +| Weak audit of “who allowed this” | Approver name stored on the deployment | +| One workflow can target any destination | Stage workflow + fixed Environment = fixed destination | + +``` +Operator clicks "Run workflow" + │ + ▼ +Job references environment: transfer-qa + │ + ▼ +GitHub pauses job ──► Slack/email to Required Reviewers + │ + ▼ +QA Approver clicks Approve (or Reject) + │ + ▼ +Environment secrets unlocked (e.g. MOSIPQA_DOCKER_TOKEN) + │ + ▼ +Reusable kattu image-transfer job runs + │ + ▼ +Deployment history + Actions log = audit trail +``` + +### Requirements / limits to know + +- **Required reviewers** and **environment secrets** on private repos need GitHub Team / Enterprise (public repos: available on Free). MOSIP public repos can use this; confirm plan if the repo is private/internal. +- Up to **6** required reviewers (users or teams). **Any one** approval is enough (not majority). +- Enable **Prevent self-review** so the person who started the run cannot approve it. +- Unapproved jobs fail after **30 days**. +- Environment secrets are only visible to jobs that set `environment: `. + +--- + +## Target setup for MOSIP image transfer + +| Environment name | Used by workflow | Required reviewers | Environment secret(s) | Fixed destination org | +|---|---|---|---|---| +| `transfer-dev2` | `image-transfer-dev2.yml` | Dev Approver team | `MOSIPDEV2_DOCKER_TOKEN` | `mosipdev2` | +| `transfer-qa` | `image-transfer-qa.yml` | QA Approver team | `MOSIPQA_DOCKER_TOKEN` | `mosipqa` | +| `transfer-prod` | `image-transfer-prod.yml` | Release admins | `MOSIPID_DOCKER_TOKEN`, `MOSIPINT_DOCKER_TOKEN` | `mosipid` / `mosipint` | + +Repo-level secrets that can stay shared (not stage-specific): `SLACK_WEBHOOK_DEVOPS`, `WIREGUARD_CONFIG`. + +--- + +## Full implementation steps + +### Phase A — Prepare people (Day 0) + +#### Step A1. Nominate roles + +| Role | Example | Count | +|---|---|---| +| Dev Transfer Operators | Engineers who prepare PRs and click Run | 2–4 | +| Dev Transfer Approvers | Dev leads who approve Environment waits | 1–3 | +| QA Transfer Operators | QA engineers | 2–4 | +| QA Transfer Approvers | QA leads | 1–3 | +| Release admins | DevOps / Release | existing | + +Approvers and Operators should be **different people** when possible. + +#### Step A2. Create GitHub teams (org settings) + +Path: GitHub **Organization** → **Teams** → **New team** + +Create: + +1. `mosip-image-transfer-dev-ops` +2. `mosip-image-transfer-dev-approvers` +3. `mosip-image-transfer-qa-ops` +4. `mosip-image-transfer-qa-approvers` +5. `mosip-release-admins` (if not already present) + +Add the named people to each team. + +#### Step A3. Grant repo access + +Path: `mosip/release-script` → **Settings** → **Collaborators and teams** + +- Give Operator teams **Write** (needed to run `workflow_dispatch`). +- Approver teams need at least **Read** (GitHub allows Environment approval with read); Write is fine if they already have it. +- Do **not** put all of Dev/QA on Write just for this — only named operators. + +--- + +### Phase B — Create Environments in the repo (UI) + +Path: `mosip/release-script` → **Settings** → **Environments** → **New environment** + +Repeat for `transfer-dev2`, `transfer-qa`, `transfer-prod`. + +#### Step B1. Create environment `transfer-dev2` + +1. Name: `transfer-dev2` +2. Click **Configure environment**. + +#### Step B2. Protection rules + +1. Check **Required reviewers**. +2. Add team: `mosip-image-transfer-dev-approvers` (or individual leads). +3. Check **Prevent self-review**. +4. Optional: **Wait timer** = `1`–`5` minutes (gives Slack time to notify). +5. Optional: **Deployment branches** → Selected branches → allow only `master` / `main` / your release branch (stops runs from random forks/branches). +6. Click **Save protection rules**. + +#### Step B3. Environment secret + +1. Under **Environment secrets** → **Add environment secret**. +2. Name: `MOSIPDEV2_DOCKER_TOKEN` +3. Value: Docker Hub / Harbor token that can **push only to `mosipdev2`**. +4. Save. + +#### Step B4. Repeat for QA and Prod + +| Environment | Reviewers | Prevent self-review | Secrets | +|---|---|---|---| +| `transfer-qa` | `mosip-image-transfer-qa-approvers` | Yes | `MOSIPQA_DOCKER_TOKEN` | +| `transfer-prod` | `mosip-release-admins` | Yes | `MOSIPID_DOCKER_TOKEN`, `MOSIPINT_DOCKER_TOKEN` | + +#### Step B5. Move tokens off repo secrets (important) + +After Environment secrets work in a test run: + +1. Settings → **Secrets and variables** → **Actions** (repository secrets). +2. Remove or rotate the old repo-level `MOSIPDEV2_DOCKER_TOKEN` / `MOSIPQA_DOCKER_TOKEN` so they are **not** usable by unprotected workflows. +3. Keep `SLACK_WEBHOOK_DEVOPS` and `WIREGUARD_CONFIG` as repository secrets if all workflows need them. + +Until you remove repo copies, a workflow **without** an Environment can still use the old repo secret — that defeats the gate. + +--- + +### Phase C — Add stage workflows (code) + +Add separate workflows so destination cannot be mistyped. + +#### Example: Dev2 workflow + +File: `.github/workflows/image-transfer-dev2.yml` + +```yaml +name: Transfer images to mosipdev2 + +on: + workflow_dispatch: + inputs: + USERNAME: + description: 'Registry username (Docker Hub user or Harbor robot)' + required: true + type: string + REGISTRY_URL: + description: 'Destination registry URL' + required: true + default: 'https://index.docker.io/v1/' + type: string + REGISTRY_TYPE: + description: 'Destination registry type' + required: true + default: 'dockerhub' + type: choice + options: + - dockerhub + - harbor + - other + ENABLE_WIREGUARD: + description: 'Enable WireGuard VPN for private Harbor' + required: false + default: false + type: boolean + +jobs: + # Gate: waits for Environment approval; unlocks Environment secrets + Image-transfer: + environment: transfer-dev2 + uses: mosip/kattu/.github/workflows/image-transfer.yml@master + with: + DESTINATION_ORGANIZATION: mosipdev2 # FIXED — operators cannot change this + REGISTRY_URL: ${{ inputs.REGISTRY_URL }} + REGISTRY_TYPE: ${{ inputs.REGISTRY_TYPE }} + ENABLE_WIREGUARD: ${{ inputs.ENABLE_WIREGUARD }} + USERNAME: ${{ inputs.USERNAME }} + secrets: + # From Environment secret (available only after approval) + TOKEN: ${{ secrets.MOSIPDEV2_DOCKER_TOKEN }} + # From repository secrets + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_DEVOPS }} + WIREGUARD_CONFIG: ${{ secrets.WIREGUARD_CONFIG }} +``` + +#### Example: QA workflow + +File: `.github/workflows/image-transfer-qa.yml` + +Same shape, but: + +- `environment: transfer-qa` +- `DESTINATION_ORGANIZATION: mosipqa` +- `TOKEN: ${{ secrets.MOSIPQA_DOCKER_TOKEN }}` + +#### Example: Prod workflow (DevOps only) + +- `environment: transfer-prod` +- Input choice limited to `mosipid` / `mosipint` only +- Tokens from prod Environment secrets +- Optionally restrict who can see/run it via repo permissions + admin-only kattu protection + +#### What to do with the old generic workflow + +| Option | Recommendation | +|---|---| +| A. Restrict to admins / leave as break-glass | Short term OK | +| B. Delete after stage workflows are proven | Preferred long term | +| C. Point it at `transfer-prod` only | If you still need one flexible entry for Release | + +Do **not** leave the old workflow able to read Dev2/QA tokens from repository secrets. + +--- + +### Phase D — Image list process (PR gate) + +Environments gate **execution**. PRs gate **what** gets transferred. + +1. Operator updates `release/vidivi/images.txt` (or split files `images-dev2.txt` / `images-qa.txt`) via PR. +2. Approver reviews image list + ticket link. +3. Merge PR. +4. Operator runs the stage workflow on the branch that contains the merged list. +5. Environment Approver approves the waiting deployment. + +Optional CODEOWNERS: + +``` +# .github/CODEOWNERS +/release/vidivi/images-dev2.txt @mosip/mosip-image-transfer-dev-approvers +/release/vidivi/images-qa.txt @mosip/mosip-image-transfer-qa-approvers +``` + +--- + +### Phase E — Operator & Approver runbook + +#### Operator (start transfer) + +1. Ensure PR with image list is merged and ticket is open. +2. Go to **Actions** → **Transfer images to mosipdev2** (or QA). +3. **Run workflow** → fill username / registry → Run. +4. Job shows status **Waiting** (yellow) for Environment approval. +5. Notify Approver (Slack) with Actions run URL + ticket. +6. After approval, wait for success; attach `transfer_report` / run URL on ticket. + +#### Approver (approve or reject) + +1. Open the Actions run URL (or repo **Deployments**). +2. Review: actor, branch, destination Environment, linked ticket/PR. +3. Click **Review deployments** → select Environment → **Approve** or **Reject**. +4. If something looks wrong (wrong branch, no ticket, unexpected overwrite): **Reject** and comment on ticket. + +Screenshot path in UI: + +`Actions` → select run → banner **Review deployments** → choose environment → Approve / Reject. + +--- + +### Phase F — Verification checklist (do this once before handover) + +- [ ] Create Environments with required reviewers + prevent self-review. +- [ ] Put Docker tokens only in Environment secrets; remove from repo secrets. +- [ ] Merge stage workflow files (`*-dev2`, `*-qa`, `*-prod`). +- [ ] Test Dev2: Operator starts run → job waits → Approver approves → transfer succeeds. +- [ ] Test self-review blocked: Operator who started run cannot approve. +- [ ] Test reject path: Approver rejects → job does not transfer. +- [ ] Confirm old generic workflow cannot push to Dev2/QA (token gone or workflow restricted). +- [ ] Confirm Slack still notifies. +- [ ] Confirm Deployment history shows actor + approver. +- [ ] Train both teams with one dry-run each. + +--- + +## How secrets resolve (important detail) + +```yaml +secrets: + TOKEN: ${{ secrets.MOSIPDEV2_DOCKER_TOKEN }} +``` + +Lookup order for a job with `environment: transfer-dev2`: + +1. Environment secret `MOSIPDEV2_DOCKER_TOKEN` on `transfer-dev2` (preferred). +2. Else repository / org secret with the same name (fallback — avoid leaving these). + +Only jobs that declare that Environment get Environment secrets, and only **after** protection rules pass. That is the core security property. + +--- + +## Mapping to reusable workflow (`mosip/kattu`) + +Your caller already does: + +```yaml +uses: mosip/kattu/.github/workflows/image-transfer.yml@master +``` + +Putting `environment:` on that **caller** job is enough: + +- GitHub waits for approval **before** starting the reusable workflow. +- Caller can pass Environment secrets into `secrets: TOKEN: ...`. +- Existing `mosipid` admin protection inside `kattu` remains an extra layer for prod. + +No change is required inside `kattu` for basic Environment gating. Optional later: add source-org allowlists inside `kattu` for defense in depth. + +--- + +## Minimal vs full adoption + +| Level | What you do | Protection gained | +|---|---|---| +| **Minimal** | One Environment on existing workflow + required reviewers | Human approval before any transfer | +| **Recommended** | Stage Environments + stage workflows + Environment secrets | Approval + least-privilege tokens + fixed destination | +| **Full** | Above + CODEOWNERS + ticket SOP + remove generic workflow + weekly access review | Process + technical + audit hygiene | + +Start with **Recommended** for `transfer-dev2` and `transfer-qa`; keep prod on `transfer-prod` with Release only. + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| Job never waits for approval | Job missing `environment:` | Add `environment: transfer-dev2` on the transfer job | +| `TOKEN` empty / unauthorized | Secret still only on repo, or wrong name; or job has no environment | Add Environment secret; ensure job references that Environment | +| Operator can approve own run | Prevent self-review off | Enable on Environment | +| Approver cannot see Approve button | Not in required reviewers / no repo access | Add to reviewer team; grant Read+ | +| Wrong images transferred | `images.txt` not reviewed | Enforce PR + CODEOWNERS before run | +| Old workflow still pushes | Repo secret still present | Delete/rotate repo secret | + +--- + +## Related docs + +- [Image Transfer Handover Plan](./image-transfer-handover-plan.md) +- [Test approval on WireGuard onboard/offboard (recommended first)](./wg-lifecycle-approval-test.md) +- [Vidivi README](../vidivi/README.md) +- GitHub docs: [Managing environments](https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/manage-environments) +- GitHub docs: [Reviewing deployments](https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/review-deployments) diff --git a/release/docs/image-transfer-handover-plan.md b/release/docs/image-transfer-handover-plan.md new file mode 100644 index 00000000..739cbc6e --- /dev/null +++ b/release/docs/image-transfer-handover-plan.md @@ -0,0 +1,344 @@ +# Image Transfer Handover Plan + +## Goal + +Hand over stage-to-stage Docker image transfer (`mosipdev → mosipdev2`, `mosipdev2 → mosipqa`, and similar paths) from DevOps to owning teams (Dev, QA), while keeping every transfer **authorized, audited, and hard to misuse**. + +This plan builds on the existing Vidivi tool and the [Manual workflow to transfer images](https://github.com/mosip/release-script/actions/workflows/image-transfer.yml). It does **not** remove DevOps ownership of production (`mosipid` / `mosipint`) or Security signing. + +--- + +## Problem today + +| Current state | Risk after naive handover | +|---|---| +| DevOps runs most transfers | Anyone with Actions access can overwrite tags | +| One workflow for all destinations | Wrong org / wrong tag / wrong version chaos | +| Secrets exist per org, but one shared workflow | Token selection mistakes; over-privilege | +| Ticket + PR practice is informal | Incomplete audit of who moved what and why | +| `mosipid` is admin-protected (via `kattu`) | Lower stages (`dev2`, `qa`) have weaker gates | + +Without controls, handover creates: accidental overwrites, tag pollution, environment skew (QA testing wrong digests), blame ambiguity, and pressure to “just re-run” without review. + +--- + +## Design principles + +1. **Least privilege** — A person or bot gets push rights only to the destination org(s) they own. +2. **Separation of duties** — Requester ≠ Approver ≠ (for prod) Releaser / Signer. +3. **Allowed paths only** — Transfers must follow the documented lifecycle; arbitrary org-to-org is blocked. +4. **PR is the request; workflow is the execution** — Image list changes are reviewed before push. +5. **Every run is attributable** — Actor, ticket, images, digests, destination, result are recorded. +6. **Fail closed** — Missing approval, wrong path, or mismatched secret → no transfer. +7. **DevOps remains the break-glass owner** — Emergency and prod stays with Release/DevOps + Security. + +--- + +## Image lifecycle (unchanged) + +``` +mosipdev → mosipdev2 → mosipqa → mosipid + ↘ mosipqa / mosipint → mosipid (staged / patch path) +``` + +Parallel Inji path (if used): `injistackdev → injistackdev2 → injistackqa → injistack`. + +| Org | Purpose | Post-handover owner | +|---|---|---| +| `mosipdev` | CI-built / early Dev | Dev (write via CI; not via transfer workflow) | +| `mosipdev2` | Pre-QA staging | Dev leads / designated Dev transfer operators | +| `mosipqa` | QA test images | QA leads / designated QA transfer operators | +| `mosipint` | Interim / patch holding | Release / DevOps (restricted) | +| `mosipid` | Community / production release | Release / DevOps + Security signing | + +--- + +## Target operating model + +### RACI (per stage) + +| Activity | Dev | QA | Release / DevOps | Security | +|---|---|---|---|---| +| Build & push to `mosipdev` (CI) | **R** | C | C | I | +| Transfer `mosipdev → mosipdev2` | **R** | C | A (policy) | I | +| Transfer `mosipdev2 → mosipqa` | C | **R** | A (policy) | I | +| Transfer `mosipqa → mosipid` / `mosipint` | C | C | **R** | **A** (signing) | +| Image signing after prod transfer | I | I | C | **R** | +| Break-glass / emergency transfer | C | C | **R** | A | +| Audit & access reviews | C | C | **R** | C | + +R = Responsible, A = Accountable, C = Consulted, I = Informed. + +### Named roles (not whole teams) + +Do **not** grant transfer rights to every Dev or QA engineer. + +| Role | Who | Rights | +|---|---|---| +| **Dev Transfer Operator** | 2–4 named Dev engineers (rotate) | Propose + (after approval) run transfers **into `mosipdev2` only** | +| **Dev Transfer Approver** | Dev lead(s) / module owners | Approve PRs / Environment for `→ mosipdev2` | +| **QA Transfer Operator** | 2–4 named QA engineers | Propose + run transfers **into `mosipqa` only** | +| **QA Transfer Approver** | QA lead(s) | Approve PRs / Environment for `→ mosipqa` | +| **Release Operator** | Build & Release / DevOps | Transfers into `mosipid` / `mosipint`; break-glass | +| **Security Signer** | Security team | Signing ticket after prod images land | +| **Audit reviewer** | Someone **not** who ran the transfer | Spot-check reports / digests (already aligned with post-release checks SoD) | + +--- + +## Allowed transfer matrix (hard rule) + +Only these destination hops are allowed for non-DevOps operators: + +| Source org (in `images.txt`) | Destination org | Who may execute | Approval required | +|---|---|---|---| +| `mosipdev` | `mosipdev2` | Dev Transfer Operator | Dev Approver | +| `mosipdev2` | `mosipqa` | QA Transfer Operator | QA Approver | +| `mosipqa` | `mosipid` | Release / DevOps (admin) | Release lead + Security signing | +| `mosipqa` / `mosipdev2` | `mosipint` | Release / DevOps (admin) | Release lead | +| Any → any other | — | **Blocked** | — | + +Additional rules: + +- Destination tag should match release/sprint convention (no free-form `latest` for MOSIP services unless explicitly allowed). +- Overwriting an existing tag in `mosipqa` / `mosipid` requires explicit approval comment (“overwrite intentional: reason”). +- Source image must exist; `check` mode must pass before `push`. +- Prefer digest verification (`hash`) when replacing a tag. + +--- + +## Control layers (defense in depth) + +Implement all layers. No single control is enough. + +### 1. People & process + +1. **Ticket required** — Jira/DSD ticket with: source org, dest org, version/tag, image list (or link to PR), reason, requester, target date. +2. **PR required** — Change `images.txt` (or a stage-specific list file) via PR. No direct push to protected branches. +3. **CODEOWNERS** — Stage-specific owners review image-list PRs. +4. **Runbook** — Operators follow a short checklist (below); free-form local `vidivi.py push` to shared orgs is discouraged / forbidden for operators without break-glass. + +### 2. GitHub repository access + +| Control | Recommendation | +|---|---| +| Who can run Actions | Restrict `workflow_dispatch` via GitHub Environments (not “all write collaborators”) | +| Branch protection | Require PR + approvals on the branch that holds image lists | +| CODEOWNERS | `/release/vidivi/images-dev2.txt` → Dev leads; `/release/vidivi/images-qa.txt` → QA leads | +| Admin list | Keep `mosipid` admin-only protection in `mosip/kattu` | + +### 3. Split workflows by stage (recommended) + +Replace “one mega-workflow anyone can aim at any org” with stage-scoped workflows: + +| Workflow | Destination fixed to | Secret | Environment | +|---|---|---|---| +| `image-transfer-dev2.yml` | `mosipdev2` only | `MOSIPDEV2_DOCKER_TOKEN` | `transfer-dev2` | +| `image-transfer-qa.yml` | `mosipqa` only | `MOSIPQA_DOCKER_TOKEN` | `transfer-qa` | +| `image-transfer-prod.yml` | `mosipid` / `mosipint` | `MOSIPID_*` / `MOSIPINT_*` | `transfer-prod` (admins + required reviewers) | + +Benefits: + +- Operators cannot “accidentally” select `MOSIPID_DOCKER_TOKEN`. +- Environment **required reviewers** gate the run. +- Audit logs clearly show which stage workflow ran. + +Keep the existing generic workflow for DevOps break-glass only, or remove it after migration. + +### 4. GitHub Environments (approval gates) + +For each environment (`transfer-dev2`, `transfer-qa`, `transfer-prod`): + +- **Required reviewers**: Approver role only (not the same GitHub group as Operators if possible). +- **Deployment branches**: Limit to agreed branches (e.g. `master` / release branch). +- **Wait timer** (optional for QA/prod): short delay so Slack notice can be seen before run proceeds. +- **Secrets scoped to Environment** — move Docker tokens from repo secrets to Environment secrets so only that workflow environment can use them. + +**Full explanation and UI/YAML steps:** [GitHub Environments for Image Transfer](./github-environments-image-transfer.md). + +### 5. Registry credentials (strongest technical gate) + +| Destination | Credential type | Scope | Who holds it | +|---|---|---|---| +| `mosipdev2` | Docker Hub PAT / Harbor robot | **Push only to `mosipdev2`** | Environment `transfer-dev2` | +| `mosipqa` | Separate token/robot | **Push only to `mosipqa`** | Environment `transfer-qa` | +| `mosipid` / `mosipint` | Separate tokens | Push to those orgs only | Environment `transfer-prod` + admin | + +Rules: + +- Never share personal Docker passwords; use org tokens / Harbor robots with minimal scope. +- Rotate tokens on operator offboarding and on a fixed schedule (e.g. quarterly). +- Disable write access for tokens used only for CI pull if any. +- Prefer **immutable tags** or digest pinning in deploy configs so a mistaken re-push is less damaging (longer-term). + +### 6. Workflow / `kattu` enforcement + +Extend reusable workflow protections beyond `mosipid`: + +| Check | Behavior | +|---|---| +| Allowed destination list per workflow | Fail if input ≠ fixed dest org | +| Allowed source org prefix in `images.txt` | Fail if source not in allowlist for that hop | +| Actor allowlist (optional) | Fail if GitHub actor not in Operators team | +| Protected orgs | Keep admin-only for `mosipid` (and add `mosipint`) | +| Dry-run / check-first | Always run `check` (and optionally `hash`) before `push` | +| Block `custom` secret path for stage workflows | No custom secret on Dev/QA workflows | + +### 7. Monitoring & audit + +Every transfer must produce a durable record: + +| Signal | Where | Purpose | +|---|---|---| +| Workflow run | GitHub Actions run history | Who triggered, inputs, success/fail | +| Environment approval | GitHub Environment deployment log | Who approved | +| PR + ticket link | PR description / commit | Why | +| `transfer_report.md` | Committed or uploaded as workflow artifact | What moved | +| Digests | Report / `hash` output | Prove exact bits | +| Slack | Channel per stage (not only DevOps) | Real-time awareness | +| Weekly digest | Automated summary of transfers | Spot unusual volume / overwrites | + +**Minimum Slack payload:** actor, ticket/PR, source→dest, image count, success/fail, run URL. + +**Retention:** Keep Actions logs and transfer reports for at least one release cycle (prefer longer for prod). + +### 8. Verification & separation of duties + +- After `→ mosipqa`, a **QA engineer who did not run the transfer** confirms sample digests / smoke pull. +- After `→ mosipid`, continue existing Security signing ticket + post-release checks by a non-releaser (`release/checks`). + +--- + +## Standard operating procedure (SOP) + +### A. Dev → Dev2 (`mosipdev → mosipdev2`) + +1. Dev Transfer Operator opens / updates ticket: list of images + tags + reason. +2. Operator opens PR updating the Dev2 image list (e.g. `images-dev2.txt`). +3. Dev Approver reviews: correct sources, tags, no unintended overwrites. +4. PR merges. +5. Operator starts `image-transfer-dev2` workflow (branch with merged list). +6. Environment `transfer-dev2` requires Dev Approver approval. +7. Workflow runs `check` → `push`; publishes report + Slack to `#image-transfer-dev2` (or equivalent). +8. Operator links run URL + report back on the ticket; closes ticket. + +### B. Dev2 → QA (`mosipdev2 → mosipqa`) + +Same pattern with QA Operator / QA Approver / `image-transfer-qa` / `#image-transfer-qa`. + +**Entry criteria for QA handoff:** Dev confirms images validated in Dev2; ticket references sprint/release; tag set matches what QA will deploy. + +### C. QA → Prod (`mosipqa → mosipid`) + +Unchanged ownership: Release/DevOps only, admin-protected org, Security signing ticket after transfer, post-release checks by independent party. + +### D. Break-glass + +1. Only Release/DevOps. +2. Ticket marked `break-glass` with incident link. +3. Use prod or break-glass workflow; dual acknowledgment in Slack. +4. Post-incident: rotate credentials if compromise suspected; write short RCA. + +--- + +## Anti-chaos / anti-misuse rules + +| Rule | Why | +|---|---| +| Named operators only (small set) | Avoid “everyone can push” | +| Approver ≠ Operator for the same run | SoD | +| Fixed destination per workflow | No org mix-ups | +| No local push with shared org tokens | Tokens stay in CI Environments | +| One ticket / one PR / one hop | Prevent mega-batches mixing stages | +| Ban silent overwrites | Require explicit approval text | +| Rate / batch expectations | Unusual large transfers need lead approval | +| Freeze windows | Optional: no QA transfers during freeze without Release approval | +| Offboarding checklist | Remove from GitHub team + rotate token same day | + +--- + +## Implementation roadmap + +### Phase 0 — Agree ownership (no code) + +- Nominate Operators and Approvers for Dev and QA. +- Confirm Slack channels and ticket project/labels (`image-transfer-dev2`, `image-transfer-qa`). +- Document freeze/break-glass contacts. + +### Phase 1 — Process without new workflows + +- Enforce ticket + PR for all lower-stage transfers. +- Add CODEOWNERS on `release/vidivi/images.txt` (or split files). +- DevOps still executes until Phase 2; teams prepare lists themselves. +- Start Slack notification habit and ticket linkage. + +### Phase 2 — Technical gates (recommended core) + +- Split workflows: `dev2`, `qa`, `prod`. +- Create GitHub Environments with required reviewers + environment-scoped secrets. +- Move tokens off shared repo-secret usage where possible. +- Restrict Environment access to Operator GitHub teams. +- Tighten `kattu` allowlists (source org + destination org). +- Keep generic workflow admin-only or retire it. + +### Phase 3 — Observability & hygiene + +- Structured Slack notifications per stage. +- Archive transfer reports as artifacts + optional commit. +- Weekly transfer summary (Actions API or script). +- Token rotation calendar; access review every sprint or monthly. +- Optional: immutable tags / digest pinning in Helm values for QA/prod. + +### Phase 4 — Continuous improvement + +- Metrics: failed transfers, overwrite rate, time-to-approve, transfers without ticket. +- Periodic drill: revoke an operator; confirm they cannot approve or run. +- Align Inji / other org paths to the same model. + +--- + +## Suggested GitHub team layout + +| GitHub team | Members | Used for | +|---|---|---| +| `mosip-image-transfer-dev-ops` | Dev Operators | Can start `transfer-dev2` | +| `mosip-image-transfer-dev-approvers` | Dev Approvers | Environment reviewers for `transfer-dev2` | +| `mosip-image-transfer-qa-ops` | QA Operators | Can start `transfer-qa` | +| `mosip-image-transfer-qa-approvers` | QA Approvers | Environment reviewers for `transfer-qa` | +| `mosip-release-admins` | Release/DevOps | `transfer-prod` + break-glass | + +Prefer **disjoint** ops vs approver teams so the same person is not always self-approving. If headcount is small, allow self-approve only for Dev2, never for QA→Prod; still keep a second person review on the PR. + +--- + +## Checklist before declaring handover complete + +- [ ] Operators and Approvers named and documented +- [ ] Stage workflows live with fixed destinations +- [ ] Environment required reviewers configured +- [ ] Tokens scoped per org and stored as Environment secrets +- [ ] CODEOWNERS / PR review on image lists +- [ ] Slack alerts working per stage +- [ ] SOP published; teams trained on one dry-run each +- [ ] `mosipid` / `mosipint` still admin-only; Security signing unchanged +- [ ] Break-glass procedure written and tested once +- [ ] Offboarding + token rotation procedure owned by DevOps +- [ ] First two real transfers audited end-to-end (ticket → PR → approve → run → report) + +--- + +## What DevOps still owns after handover + +- Production / interim transfers (`mosipid`, `mosipint`) +- Reusable workflow protections in `mosip/kattu` +- Registry org policy, token issuance/rotation +- Break-glass execution +- Access reviews and audit tooling +- Platform changes to Vidivi / Actions + +Dev and QA own **their hop only**: preparing the image list, getting approval, triggering the stage workflow, and confirming the result for that stage. + +--- + +## Summary + +Hand over **authority by stage**, not the whole transfer button. Combine **named operators**, **mandatory ticket+PR**, **stage-specific workflows**, **GitHub Environment approvals**, **org-scoped registry tokens**, and **Slack/report audit**. Keep **prod and signing** with Release/DevOps + Security. That is the clean path to remove DevOps from day-to-day `dev→dev2` and `dev2→qa` moves without inviting misuse or chaos. diff --git a/release/docs/wg-lifecycle-approval-test.md b/release/docs/wg-lifecycle-approval-test.md new file mode 100644 index 00000000..7c870266 --- /dev/null +++ b/release/docs/wg-lifecycle-approval-test.md @@ -0,0 +1,129 @@ +# Test GitHub Environment approval on WireGuard onboard/offboard + +Use the rapid-deployment **WireGuard onboard/offboard** workflow in +[`mosip/infra`](https://github.com/mosip/infra/blob/master/.github/workflows/wg-onboard.yml) +as the first place to try Approve / Reject — before rolling the same pattern +to image transfer. + +## Important design rule + +Do **not** set: + +```yaml +environment: ${{ inputs.ENV_NAME }} +``` + +That input is the **target** Environment this workflow creates (Helmsman secrets +`TF_WG_CONFIG`, `CLUSTER_WIREGUARD_WG0`, `CLUSTER_WIREGUARD_WG1`). On first +onboard it often does not exist yet, so it cannot be the approval gate. + +Use a **fixed gate** Environment instead: + +```yaml +jobs: + onboard: + environment: wg-lifecycle # approval gate (fixed name) +``` + +| Name | Role | +|---|---| +| `wg-lifecycle` | Who must Approve before the job runs | +| `inputs.ENV_NAME` (e.g. `qa-foo`) | Where WG peer secrets are written | + +## Step 1 — Create Environment `wg-lifecycle` (UI, ~2 minutes) + +In **https://github.com/mosip/infra**: + +1. **Settings** → **Environments** → **New environment** +2. Name: `wg-lifecycle` (exact spelling) +3. Enable **Required reviewers** → add yourself + one other person (or a team) +4. Enable **Prevent self-review** +5. Optional: wait timer `1` minute; limit deployment branches to `master` +6. **Save protection rules** + +You do **not** need to move secrets yet for the first UI test. + +## Step 2 — One-line workflow change + +In `.github/workflows/wg-onboard.yml`, under `jobs.onboard`, add: + +```yaml +jobs: + onboard: + runs-on: [self-hosted, Linux, X64] + environment: wg-lifecycle # ← add this line + timeout-minutes: 20 + steps: + # ... existing steps unchanged ... +``` + +Commit on a branch and merge (or test from that branch via **Run workflow** → select the branch). + +Full proposed file comment header: + +```yaml +# Approval gate: job uses GitHub Environment "wg-lifecycle" (required reviewers). +# That gate is SEPARATE from the target ENV_NAME environment this workflow creates. +``` + +## Step 3 — Safe test run (DRY_RUN = true) + +1. **Actions** → **WireGuard environment onboard/offboard** → **Run workflow** +2. Use: + +| Input | Value | +|---|---| +| Branch | branch that contains `environment: wg-lifecycle` | +| `ACTION` | `onboard` | +| `ENV_NAME` | e.g. `wg-approval-test` | +| `JUMPSERVER_HOST` | your usual jumpserver | +| `TICKET` | `TEST-APPROVAL` | +| `DRY_RUN` | **true** (default) | + +3. Open the run → job should be **Waiting** on `wg-lifecycle` +4. As **Approver** (not the person who clicked Run): **Review deployments** → Approve +5. Job continues; dry-run logs show planned actions without writing peers/secrets +6. Repeat once with **Reject** to confirm the deny path + +## Step 4 — What you should see + +| Checkpoint | Expected | +|---|---| +| Before approve | Job status Waiting; no SSH/script steps yet | +| After approve | Steps run; deployment shows actor + approver | +| Self-review | Operator cannot approve own run | +| Reject | Job does not perform onboard | +| DRY_RUN=true | No secret/`assigned.txt` writes | + +## Step 5 — Optional hardening (after UX works) + +1. Copy `ACTION_PAT` and `MOSIP_AWS_PEM` to **Environment secrets** on `wg-lifecycle` +2. Remove them from repository secrets (so only approved jobs can use them) +3. Only then use `DRY_RUN=false` for a real onboard with a real ticket + +## Day-to-day after rollout + +| Who | Does | +|---|---| +| QA/Dev operator | Runs workflow with ticket + ENV_NAME | +| DevOps / lead approver | Approves or rejects waiting deployment | +| Same person for both | Blocked if Prevent self-review is on | + +## Same pattern for image transfer later + +| Action | Gate Environment | Fixed destination | +|---|---|---| +| WG onboard/offboard | `wg-lifecycle` | N/A (target is `ENV_NAME` input) | +| Images → mosipdev2 | `transfer-dev2` | `mosipdev2` | +| Images → mosipqa | `transfer-qa` | `mosipqa` | + +WireGuard is the better first test because **dry-run is already the default**. + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| No Waiting state | `environment:` missing on that branch | Merge/select correct branch | +| Waiting but no reviewers notified | Required reviewers not saved | Re-open Environment settings | +| Operator can approve self | Prevent self-review off | Enable it | +| Job waits forever | Approver not in list / no access | Add reviewer; grant at least Read on repo | diff --git a/release/vidivi/README.md b/release/vidivi/README.md index dc60d521..92e776ec 100644 --- a/release/vidivi/README.md +++ b/release/vidivi/README.md @@ -4,6 +4,8 @@ ## Overview +For handing stage transfers (`mosipdev` → `mosipdev2`, `mosipdev2` → `mosipqa`) from DevOps to Dev/QA with access control, approvals, and monitoring, see **[Image Transfer Handover Plan](../docs/image-transfer-handover-plan.md)** and **[GitHub Environments setup (full steps)](../docs/github-environments-image-transfer.md)**. + In MOSIP, we maintain several Docker Hub organizations with specific purposes: * **`mosipid`**: Contains officially released Docker images for the Open Source Community