🛡️ DevSecOps GitOps Platform

A production-shaped, security-first Kubernetes platform — 100% GitOps, zero-downtime automated blue-green delivery, cryptographically-verified supply chain, and full-stack observability.

---

📖 What is this?

This repository is the single source of truth for an end-to-end DevSecOps platform running on Kubernetes. Every workload, policy, dashboard, and route in the cluster is declared here as code — a human never runs kubectl apply. Argo CD continuously reconciles the live cluster to match main, so the repository is the deployment.

The platform demonstrates the full DevSecOps lifecycle on real infrastructure:

Commit → CI build → keyless image signing → GitOps sync → policy-gated admission → automated smoke-tested blue-green promotion → runtime detection → full observability.

It is deliberately built the way a real platform team would build one: declarative, auditable, secure by default, and self-healing.

---

✨ Highlights

Capability	How it's implemented
🔁 Pure GitOps	Argo CD app-of-apps — one root Application fans out to 11 self-healing Applications. Auto-sync, auto-prune, drift correction.
🚀 Zero-downtime delivery	Argo Rollouts blue-green with fully automated promotion — measured 0 failed requests across a live cutover.
🧪 Automated release gates	Smoke tests (AnalysisTemplate) run against the green environment before traffic flips, and against production after — with automatic rollback.
🔐 Verified supply chain	Sigstore cosign keyless signing in CI + Kyverno verifyImages admission policy. Unsigned or tampered images are rejected at the API server — even with valid registry credentials.
📜 Policy as Code	Kyverno enforces image provenance; the framework is in place to extend to Pod Security, resource limits, and registry allow-lists.
🛰️ Runtime security	Falco (eBPF) detects suspicious syscalls in real time → Falcosidekick → web UI.
📊 Full-stack observability	Prometheus (metrics) + Loki (logs via Grafana Alloy) + Grafana (unified UI) + Alertmanager.
🔑 Secret hygiene	No credential ever lives in git. All secrets are Kubernetes Secrets referenced by name; charts consume them via existingSecret.
🌐 Unified HTTPS ingress	Every UI served on 443 through Traefik with path- and host-based routing.

---

🏗️ Architecture

flowchart TB
    subgraph dev["👨‍💻 Developer"]
        code["git push"]
    end

    subgraph ci["⚙️ CI — GitHub Actions (devsecops-pipeline)"]
        scan["Scan gates<br/>ESLint · npm audit<br/>Gitleaks · SonarCloud"] --> build["Build image"]
        build --> imgscan["Trivy CVE<br/>+ Syft SBOM"] --> push["Push to Docker Hub"]
        push --> sign["cosign keyless sign<br/>+ Rekor transparency log"]
    end

    subgraph git["📦 GitOps Repo (this repo — source of truth)"]
        root["root-app"] --> apps["11 Argo CD Applications"]
    end

    subgraph cluster["☸️ k3s Cluster"]
        direction TB
        argo["Argo CD<br/>(reconcile loop)"]
        subgraph delivery["Progressive Delivery"]
            rollout["Argo Rollouts<br/>blue-green + smoke tests"]
            kyverno["Kyverno<br/>verifyImages admission"]
        end
        subgraph sec["Security"]
            falco["Falco (eBPF)<br/>runtime detection"]
        end
        subgraph obs["Observability"]
            prom["Prometheus"]
            loki["Loki + Alloy"]
            graf["Grafana"]
        end
        app["cowsay app<br/>(blue / green)"]
    end

    code --> build
    push -.image.-> app
    code --> git
    git --> argo
    argo --> delivery & sec & obs
    kyverno -->|verify signature| app
    rollout -->|promote| app
    falco -.watches.-> app
    obs -.observe.-> app

    classDef green fill:#2f7e3f,stroke:#1b5e20,color:#fff
    classDef blue fill:#1b98e0,stroke:#0d47a1,color:#fff
    class sign,kyverno,falco green
    class argo,rollout blue

Loading

---

🧰 Technology Stack

Layer	Tools
Infrastructure	GCP Compute Engine · k3s (lightweight Kubernetes)
GitOps / CD	Argo CD (app-of-apps) · Argo Rollouts (blue-green)
CI Security Scanning	ESLint (SAST) · npm audit (SCA) · Gitleaks (secrets) · SonarCloud (code quality/SAST) · Trivy (image CVEs) · Syft (SBOM)
CI / Supply Chain	GitHub Actions · Sigstore cosign (keyless OIDC) · Rekor transparency log · Docker Hub
Policy & Admission	Kyverno — verifyImages, ClusterPolicy
Runtime Security	Falco (modern eBPF) · Falcosidekick + UI
Metrics	Prometheus · Alertmanager · kube-state-metrics · node-exporter
Logs	Loki · Grafana Alloy (log collector)
Dashboards	Grafana (Prometheus + Loki + Alertmanager datasources)
Ingress / TLS	Traefik (path + host routing on 443)
Packaging	Helm · Kustomize (base + overlays)

---

⚙️ CI Pipeline — shift-left security gates

The application source lives in a separate repo (devsecops-pipeline) whose GitHub Actions workflow enforces security before an image is ever published. Every push to main runs a fail-fast gauntlet — if any gate fails, no image is built, pushed, or signed. Pull requests run the same scans (build + test) but skip publish/sign, so untrusted code can never produce a signed artifact.

flowchart LR
    A["ESLint<br/><sub>SAST · lint</sub>"] --> B["npm audit<br/><sub>SCA · deps</sub>"]
    B --> C["Gitleaks<br/><sub>secret scan</sub>"]
    C --> D["SonarCloud<br/><sub>code quality</sub>"]
    D --> E["Docker Build<br/><sub>sha + latest</sub>"]
    E --> F["Trivy<br/><sub>CVE scan</sub>"]
    F --> G["Syft<br/><sub>SBOM</sub>"]
    G --> H["Docker Hub<br/><sub>push</sub>"]
    H --> I["Cosign<br/><sub>keyless sign</sub>"]

    classDef gate fill:#b3261e,stroke:#7f1d1d,color:#fff
    classDef build fill:#1b98e0,stroke:#0d47a1,color:#fff
    classDef publish fill:#2f7e3f,stroke:#1b5e20,color:#fff
    class A,B,C,D,F gate
    class E,G build
    class H,I publish

Loading

#	Stage	Category	What it does / why it's a gate
1	ESLint (--max-warnings=0)	SAST / lint	Fails on any lint warning — code style and correctness enforced with zero tolerance.
2	npm audit (--audit-level=high)	SCA	Fails the build on high/critical vulnerabilities in dependencies.
3	Gitleaks	Secret scanning	Scans the full history for leaked credentials, tokens, and keys.
4	SonarCloud	SAST / quality	Deep static analysis — bugs, code smells, security hotspots, coverage.
5	Docker Build	Packaging	Builds and tags the image with the immutable git SHA and latest.
6	Trivy (exit-code: 1, CRITICAL,HIGH)	Image scanning	Fails the build if the built image contains critical/high CVEs.
7	Syft	SBOM	Generates an SPDX SBOM of the image, uploaded as a build artifact.
8	Docker Hub push (push only)	Publish	Pushes sha + latest tags — reached only after every gate is green.
9	Cosign sign (push only)	Supply chain	Keyless signs both tags via GitHub OIDC → the signature Kyverno verifies at admission (next section).

Least-privilege permissions: the workflow requests only contents: read, security-events: write, actions: read, and id-token: write (the last enables keyless OIDC signing — no long-lived signing key exists).

This CI job produces the signature that the platform verifies at deploy time — the two halves meet at the Kyverno admission gate below. 👇

---

🔐 Supply-Chain Security — the trust chain

This is the heart of the "Sec" in DevSecOps. It uses keyless signing — there is no private key to steal; trust is rooted in identity.

sequenceDiagram
    participant CI as GitHub Actions (CI)
    participant Fulcio as Sigstore Fulcio (CA)
    participant Rekor as Rekor (transparency log)
    participant Reg as Docker Hub
    participant K8s as Kubernetes API
    participant Kyverno as Kyverno Webhook

    CI->>Fulcio: OIDC token "I am CI.yml @ main"
    Fulcio-->>CI: short-lived signing certificate (~10 min)
    CI->>Reg: push image + signature
    CI->>Rekor: record signature (public, immutable)
    Note over K8s,Kyverno: later, at deploy time
    K8s->>Kyverno: admission review (new Pod)
    Kyverno->>Reg: fetch signature
    Kyverno->>Rekor: verify inclusion
    Kyverno-->>K8s: ✅ issuer + subject match → admit<br/>❌ otherwise → REJECT

Loading

Kyverno policy (Enforce mode) requires every Pod to carry a cosign signature proving it was built by exactly the GitHub Actions workflow CI.yml on main of the pipeline repo, verified against the Rekor transparency log — then rewrites the tag to an immutable digest (mutateDigest) so what was verified is what runs. An attacker with stolen registry credentials still cannot run a tampered image.

---

🚀 Zero-Downtime Blue-Green Delivery

The application is an Argo Rollouts Rollout with a fully automated blue-green strategy.

flowchart LR
    A["New image<br/>synced by Argo CD"] --> B["🟢 Green ReplicaSet<br/>starts (users still on 🔵 Blue)"]
    B --> C{"Pre-promotion<br/>smoke tests<br/>(preview URL)"}
    C -- "❌ fail" --> X["Abort — users never affected"]
    C -- "✅ pass" --> D["⚡ Traffic flips to Green<br/>(instant, zero downtime)"]
    D --> E{"Post-promotion<br/>smoke tests<br/>(live URL)"}
    E -- "❌ fail" --> R["⏪ Auto-rollback to Blue<br/>(kept warm 30s)"]
    E -- "✅ pass" --> F["🟢 Green becomes new Blue"]

Loading

• No human in the loop — from image push to production cutover is fully automated.
• The gate is real — a broken build fails its smoke tests on the preview environment and never reaches users.
• Verified zero-downtime — a probe fired continuously at cowsay.danindu.site during a real cutover recorded 75/75 successful requests.
• Live and preview — production traffic serves the active (blue) version at cowsay.danindu.site; the next (green) version is exercised on a separate preview host before it is ever promoted.

# Watch a release happen live
kubectl argo rollouts get rollout devsecops -n devsecops --watch

---

📊 Observability

A complete metrics-and-logs stack, all reconciled by GitOps:

• Prometheus scrapes the cluster and workloads (bounded retention for a lean footprint).
• Grafana Alloy ships every pod's logs to Loki.
• Grafana ships pre-wired with Prometheus, Loki, and Alertmanager datasources — metrics and logs in one pane.
• Falco streams runtime security events to the Falcosidekick UI (e.g. "Terminal shell in a container" the instant someone execs into a pod).

---

🗂️ Repository Structure

.
├── argocd/
│   ├── root-app.yaml                 # the ONE Application applied by hand — bootstraps everything
│   └── apps/                         # app-of-apps: what Argo CD manages
│       ├── devsecops-app.yaml        #   → the application workload
│       └── platform-apps.yaml        #   → the platform (multi-source)
│
├── apps/devsecops/                   # the application (Kustomize)
│   ├── base/
│   │   ├── rollout.yaml              # Argo Rollouts blue-green Rollout
│   │   ├── analysis-template.yaml    # automated smoke tests (release gate)
│   │   ├── service.yaml              # active (blue) service
│   │   └── service-preview.yaml      # preview (green) service
│   └── overlays/prod/                # environment-specific image pin
│
└── platform/                         # the platform components
    ├── kyverno/                      # policy engine (Helm)
    ├── kyverno-policy/               # cosign image-verification ClusterPolicy
    ├── observability/                # kube-prometheus-stack · Loki · Alloy (Helm + values)
    ├── falco/                        # runtime security (Helm, eBPF)
    ├── argo-rollouts/               # progressive delivery controller (Helm)
    └── ingress/                      # Traefik routing for every service

---

⚡ How It Works (GitOps reconciliation)

1. Bootstrap once: kubectl apply -f argocd/root-app.yaml.
2. The root Application watches argocd/apps/ and creates every child Application.
3. Each child Application deploys its slice of the platform — from Helm registries or from this repo.
4. Everything self-heals: manual cluster changes are automatically reverted to match git. Merging to main is deploying.

# One command reconstructs the entire platform on a fresh cluster:
kubectl apply -f argocd/root-app.yaml

---

🔒 Security & Secrets Posture

• Zero secrets in git. Every credential is a Kubernetes Secret referenced by name (existingSecret); the manifests contain only references.
• Admission-time enforcement via Kyverno — bad images can't even be scheduled.
• Runtime detection via Falco eBPF — suspicious behavior is surfaced live.
• TLS everywhere — all UIs are served over HTTPS through Traefik.
• Least privilege — the workload runs as non-root, read-only root filesystem, all Linux capabilities dropped.

---

🧠 Engineering Decisions Worth Noting

• Modern eBPF for Falco instead of kernel modules — no kernel-header build step, works on managed cloud kernels.
• Grafana Alloy over Promtail — Promtail is end-of-life; Alloy is the supported successor.
• Multi-source Argo CD Applications — Helm charts from upstream registries combined with values files from this repo via $values refs, keeping upstream charts pristine.
• Server-Side Apply for large CRDs — avoids the client-side annotation size limit that breaks big policy CRDs.
• Blue-green over rolling update — enables a real, testable pre-production environment and instant rollback.

---

✅ Implemented

• Pure GitOps — app-of-apps, self-healing, auto-prune
• CI security gates — ESLint, npm audit, Gitleaks, SonarCloud, Trivy, Syft SBOM
• Keyless image signing (cosign) + admission-time verification (Kyverno)
• Immutable git-SHA image tags produced in CI
• Zero-downtime blue-green with automated smoke-test gates and rollback
• Runtime security (Falco eBPF) and full-stack observability (Prometheus/Loki/Grafana)
• Zero secrets in git — all credentials as referenced Kubernetes Secrets

---

Built with a GitOps-first, security-by-default philosophy.

Declarative · Auditable · Self-healing · Zero-downtime