GitOps-driven application deployment with ArgoCD, phased rollouts, and 60+ self-hosted services
A single Git repo that defines the entire application layer — from infrastructure controllers to user-facing services
- Overview
- GitOps Architecture
- Value Cascade Design
- Infrastructure Layers
- Service Categories
- Network & Data Flow
- Observability
- Key Design Patterns
- Helper Scripts
- CI/CD Automation
- Prerequisites
- Adding a New Service
- License & Contributing
This repository is the single source of truth for everything running on the K3s cluster. ArgoCD watches this repo and continuously reconciles the desired state defined here with the live cluster. The architecture features:
- Automatic service discovery — drop an
app.yamlfile, ArgoCD picks it up - 4-phase ordered deployment — CRDs before controllers, controllers before services
- Cascading value hierarchy — change a domain or IP once, it propagates to every service
- Base/variant inheritance — language or regional variants share 95% of their config with the base service
- Sealed Secrets in Git — encrypted secrets committed alongside manifests, decrypted in-cluster
The repo manages infrastructure controllers (Traefik, Authentik, cert-manager, databases), observability (Prometheus, Grafana, Loki, Tempo), and 60+ user-facing applications across home automation, media, and operations categories.
Two ApplicationSets at the bootstrap level drive all deployments:
The Matrix generator combines a list of categories (infrastructure, services) with a Git file scanner. Every directory containing an app.yaml is automatically discovered and deployed — no manual Application manifests needed.
Under the hood, the templatePatch resolves each discovered app.yaml into a multi-source ArgoCD Application — handling variant inheritance, value layering, and optional ingress injection — as shown in the diagram above.
Deployments are ordered through 4 phases to guarantee dependency resolution:
| Phase | Label | What Deploys | Why It's First |
|---|---|---|---|
| 1 | crds |
Prometheus Operator CRDs | CRDs must exist before any ServiceMonitor/PodMonitor |
| 2 | foundation |
cert-manager, sealed-secrets, NFS, MetalLB | TLS, secrets, storage, and load balancing are prerequisites |
| 3 | controllers |
Traefik, Authentik, databases, CrowdSec, Intel GPU | Ingress, auth, and data layers must be ready for services |
| 4 | services |
All user-facing applications | Everything they depend on is guaranteed available |
Each phase completes fully before the next begins, ensuring clean bootstrap even on a fresh cluster.
Every Helm release inherits configuration from a strict 4-layer hierarchy — maximizing reuse and minimizing duplication:
flowchart TB
G(["🌍 globals.yaml\nCluster-wide: domains, IPs,\nNFS server, storage class"])
C(["📋 common.yaml\nCategory defaults: TZ, PUID/PGID,\nprobes, default PVC template"])
S(["📦 service/values.yaml\nService-specific: image, ports,\nenv vars, persistence mounts"])
V(["🔀 variant/values.yaml\nDelta override: image tag,\nLB IP, locale settings"])
G ==>|"Layer 1"| C
C ==>|"Layer 2"| S
S ==>|"Layer 3"| V
classDef globals fill:#E57000,stroke:#CC6300,color:#fff
classDef common fill:#7B42BC,stroke:#6A35A3,color:#fff
classDef service fill:#326CE5,stroke:#2B5FC2,color:#fff
classDef variant fill:#0F1689,stroke:#0D1270,color:#fff
class G globals
class C common
class S service
class V variant
| Layer | Scope | Example |
|---|---|---|
| globals.yaml | Every release in the cluster | Domains, NFS server IP, MetalLB pool IPs, default storage class |
| common.yaml | All services in a category | Timezone, PUID/PGID, liveness/readiness probes, NFS PVC template |
| values.yaml | Single service | Container image, port mappings, environment variables, mount paths |
| variant values | Override layer (optional) | Image tag override, alternate LoadBalancer IP, locale-specific settings |
Impact: Changing the NFS server IP in globals.yaml automatically propagates to every service using NFS storage on the next sync — zero individual edits.
The infrastructure is organized into four tiers, deployed in order:
infrastructure/
├── system/ Phase 2 (Foundation)
│ ├── cert-manager TLS automation (Let's Encrypt + Cloudflare DNS-01)
│ ├── sealed-secrets In-cluster secret decryption
│ ├── nfs-provisioner Dynamic PV provisioning from TrueNAS
│ ├── metallb L2 load balancer (multiple IP pools)
│ ├── intel-device-operator GPU scheduling (SR-IOV virtual functions)
│ ├── crowdsec Intrusion detection with mTLS
│ └── prometheus-crds Phase 1 — CRD-only (no Prometheus yet)
│
├── controllers/ Phase 3 (Controllers)
│ ├── traefik DaemonSet ingress with CrowdSec bouncer
│ ├── authentik SSO provider (OIDC, LDAP, ForwardAuth)
│ └── databases PostgreSQL + Redis (shared by services)
│
├── configs/ Phase 3-4 (Configuration)
│ └── templates/ IngressRoutes, middlewares, certs, notifications
│
└── base-configs/ Phase 2 (Foundation)
└── templates/ Storage PVs, backup CronJobs (PostgreSQL, TLS certs)
Services are organized into three domains. Each service is self-contained in its own directory with an app.yaml and values.yaml:
Smart home infrastructure including a home automation platform, MQTT broker, and Zigbee coordinator bridge — working together for IoT device management across the local network.
A comprehensive media management ecosystem covering acquisition, organization, streaming, and discovery — with GPU-accelerated transcoding and multi-language support via base/variant service pairs.
The operational backbone: full observability stack (metrics, logs, traces, dashboards, alerting), document management, password vault, push notifications, and a collection of self-hosted productivity tools.
Services are intentionally described by category rather than enumerated individually — the lineup evolves continuously as services are added, removed, or upgraded.
External and internal traffic follow a layered security path; persistent data is served from TrueNAS over NFS:
| Layer | Scope | Function |
|---|---|---|
| MetalLB | All traffic | L2 load balancing to Traefik DaemonSet pods |
| Traefik | All traffic | TLS termination, IngressRoute routing |
| CrowdSec | External only | Bot detection, IP reputation, rate limiting |
| Authentik | Per-service opt-in | OIDC/ForwardAuth SSO with group-based access |
| Rate Limiter | Per-service opt-in | Request throttling (configurable per route) |
A full observability stack provides metrics, logs, traces, and alerting:
Drop an app.yaml file anywhere in the repo — the Matrix generator finds it and creates an ArgoCD Application. No boilerplate Application manifests to maintain.
Language variants (e.g., a Russian-language media manager) point to a base service via baseApp and only override what differs. A variant is typically 3-5 lines of YAML rather than duplicating an entire service config.
A custom Helm chart generates Traefik IngressRoutes from declarative app.yaml fields. One line (auth: true) wires up Authentik ForwardAuth, TLS, and CrowdSec — no per-service IngressRoute YAML to maintain.
Traefik's CrowdSec bouncer plugin is downloaded via init container to emptyDir, not fetched at runtime. This ensures cluster restarts work even if GitHub is down.
Secrets are encrypted with a pre-seeded certificate (matching the Ansible bootstrap), committed to Git as SealedSecret CRDs, and decrypted in-cluster. The full lifecycle spans three repos:
sequenceDiagram
autonumber
participant Vault as Ansible Vault
participant K8s as Kubernetes
participant Argo as ArgoCD
participant SS as Sealed Secrets<br/>Controller
participant Dev as Developer
participant Git as Apps Repo
rect rgb(238, 0, 0)
Note over Vault,K8s: Cluster Bootstrap (Ansible)
Vault->>K8s: Pre-seed TLS keypair as Secret
Note right of K8s: labeled sealed-secrets-key: active
end
rect rgb(123, 66, 188)
Note over Argo,SS: Phase 2 — Foundation
Argo->>K8s: Deploy sealed-secrets controller
SS->>K8s: Find and adopt pre-seeded key
end
rect rgb(50, 108, 229)
Note over Dev,Git: Day-to-Day Usage
Dev->>Dev: seal.sh encrypts with<br/>matching public cert
Dev->>Git: Commit SealedSecret YAML
Git-->>Argo: Sync detected
Argo->>K8s: Apply SealedSecret CRD
SS->>K8s: Decrypt → plain Secret
K8s-->>K8s: Pod mounts Secret<br/>as env or volume
end
The key insight: because Ansible pre-seeds the exact keypair that seal.sh encrypts against, secrets can be sealed and committed to Git before the cluster even exists — and they'll decrypt correctly on first boot.
Each ArgoCD Application pulls from up to 4 sources: the Helm chart repo, the Git values layers, the ingress chart, and raw manifests — composing complex deployments declaratively.
| Script | Purpose |
|---|---|
scripts/new-service.sh |
Interactive scaffolding — prompts for image, ingress, persistence and generates app.yaml + values.yaml |
scripts/seal.sh |
Encrypts secrets into SealedSecret YAML using the cluster's pre-seeded certificate |
scripts/ntfy-manager.sh |
CLI wrapper for managing ntfy users and access rules inside the running pod |
Four workflows ensure safe, validated, and secure deployments:
flowchart TD
subgraph pr["PR Phase"]
PR([Pull Request]) --> VAL[validate-and-diff.yml]
VAL --> LINT["YAML Lint +\nKubeconform"]
VAL --> DIFF>ArgoCD Diff\nPreview]
PR --> FMT[format.yaml\nPrettier formatting]
end
subgraph merge["Merge Phase"]
MERGE([Push to Main]) ==> REF[refresh.yaml]
REF ==> SCOPE{{"Scope Detection\nWhich apps changed?"}}
SCOPE ==> SYNC(["ArgoCD Sync\nOnly affected apps"])
end
subgraph sched["Scheduled Phase"]
CRON([Mondays 06:00 UTC]) ==> SCAN[image-scan.yml\nTrivy image scan]
SCAN ==> SEC>"GitHub Security tab\nCRITICAL/HIGH findings"]
end
classDef val fill:#EF7B4D,stroke:#D66A3D,color:#fff
classDef ref fill:#326CE5,stroke:#2B5FC2,color:#fff
classDef scan fill:#1904DA,stroke:#1403B0,color:#fff
class VAL val
class REF ref
class SCAN scan
| Workflow | Trigger | Purpose |
|---|---|---|
| validate-and-diff | PR | YAML lint + Kubeconform schema validation + ArgoCD diff preview |
| format | PR | Prettier formatting for YAML/JSON/shell files |
| refresh | Push to main | Smart scope detection — only syncs ArgoCD apps affected by the change |
| image-scan | Schedule (Mondays 06:00 UTC) + manual | Trivy scan of deployed images — publishes CRITICAL/HIGH findings to the GitHub Security tab |
The refresh workflow is scope-aware: it analyzes the git diff to determine whether to refresh all apps, all services, all infrastructure, or just specific applications — avoiding unnecessary reconciliation cycles.
- K3s cluster provisioned by the Ansible playbook (with ArgoCD bootstrapped)
- kubeseal CLI + sealed-secrets certificate for secret encryption
- kubectl configured with cluster access
- yq and crane (for
new-service.shscaffolding script)
# Interactive scaffolding
./scripts/new-service.sh
# Or manually create:
# services/<category>/<service-name>/app.yaml
# services/<category>/<service-name>/values.yaml
# Encrypt any secrets
./scripts/seal.sh <secret-name> <namespace>
# Commit and push — ArgoCD discovers and deploys automatically
git add . && git commit -m "feat: add <service>" && git pushThe ApplicationSet will discover the new app.yaml on the next sync cycle and create an ArgoCD Application for it automatically.
This is a personal homelab project. Feel free to use it as inspiration for your own infrastructure. If you spot an issue or have a suggestion, open an issue — contributions and feedback are welcome.