Edge-Computing-LLM · waqasm86 · Jul 17, 2026 · Jul 8, 2026 · Jul 17, 2026 · Jul 17, 2026
diff --git a/.github/workflows/validate.yaml b/.github/workflows/validate.yaml
@@ -0,0 +1,24 @@
+name: validate
+
+on:
+  push:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  go:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v7
+        with:
+          go-version-file: go.mod
+          cache: true
+      - run: go mod verify
+      - name: Check formatting
+        run: test -z "$(gofmt -l .)"
+      - run: go test ./...
+      - run: go vet ./...
+      - run: go build ./...
diff --git a/.gitignore b/.gitignore
@@ -25,6 +25,10 @@ coverage.*
 *.bak
 *.swp
 *~
+local-run-output/
+validation-output/
+evidence/
+reports/
 
 # Editor and OS metadata
 .DS_Store
@@ -35,8 +39,13 @@ Thumbs.db
 # Kubernetes and Helm local artifacts
 kubeconfig
 *.kubeconfig
+*.kubeconfig.yaml
 /.helm/
 /.kube/
+*.tgz
+*.tar
+*.tar.gz
+*.zip
 
 # Vendored dependencies are not committed unless intentionally enabled later.
 /vendor/
diff --git a/README.md b/README.md
@@ -9,8 +9,19 @@ It coordinates two current layers:
 - `llm-observability-stack`: LLMOps workloads, Helm charts, Open WebUI, Ollama,
   OpenTelemetry, Prometheus, Grafana, and related services.
 
-The CLI is intentionally NVIDIA-only. It does not include AMD, Intel, Apple
-Silicon, or CPU-only accelerator abstractions.
+The organization also publishes
+[`qwen-gguf-observability`](https://github.com/Edge-Computing-LLM/qwen-gguf-observability),
+a read-only evidence companion for the deployed Qwen runtime. It consumes the
+status produced by these layers; `edge-cli` does not install it as another
+cluster layer.
+
+[`Frontend-Edge-LLM-Observability`](https://github.com/Edge-Computing-LLM/Frontend-Edge-LLM-Observability)
+is the TypeScript/Vue presentation companion. It is also outside the ordered
+cluster lifecycle and consumes controlled metrics endpoints rather than direct
+host or Kubernetes credentials.
+
+The CLI supports automatic NVIDIA/CPU selection. It does not currently install
+AMD, Intel, or Apple Silicon accelerator runtimes.
 
 ## Why this exists
 
@@ -19,7 +30,8 @@ workloads. `edge-cli` provides one installable control plane so operators can ru
 preflight checks, installs, validation, status, logs, and safe uninstall flows
 without treating each repository as a disconnected CLI application.
 
-Future repositories can be added as new Go modules under `internal/modules`.
+Future repositories can be added as new layers in `internal/platform` and new Go
+modules under `internal/modules`.
 
 ## Install
 
@@ -51,8 +63,8 @@ Default configuration:
 
 ```yaml
 repos:
-  k3sNvidiaEdge: /media/waqasm86/External1/Waqas-Projects/Project-Edge-Computing-LLM/k3s-nvidia-edge
-  llmObservabilityStack: /media/waqasm86/External1/Waqas-Projects/Project-Edge-Computing-LLM/llm-observability-stack
+  k3sNvidiaEdge: /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/k3s-nvidia-edge
+  llmObservabilityStack: /media/waqasm86/External1/Waqas-Projects/Project-Linux-Kubernetes-Nvidia/Project-Edge-Computing-LLM/llm-observability-stack
 cluster:
   kubeconfig: ""
   defaultNamespace: llm-observability
@@ -85,20 +97,39 @@ edge repo doctor
 ## Full Install
 
 ```bash
-edge doctor
-edge install all --yes
+edge install all --accelerator auto --yes
 edge status
 ```
 
+`auto` deploys the NVIDIA infrastructure layer when `nvidia-smi` detects a
+working GPU. Without NVIDIA hardware it skips the toolkit and GPU Operator,
+installs or validates basic k3s, and selects `values.cpu-k3s.yaml`.
+
+On the validated GeForce 940M profile, the application smoke test targets
+`qwen-1-8b-chat-q4-k-m-local`; CPU and general profiles retain their configured
+Gemma model defaults.
+
+Explicit modes are useful in automation:
+
+```bash
+edge install all --accelerator nvidia --yes
+edge install all --accelerator cpu --yes
+```
+
 `edge install all` runs:
 
-1. Infra diagnostics.
-2. Linux/k3s/NVIDIA GPU infrastructure install.
-3. Infra validation and CUDA validation pod.
+1. Host accelerator detection.
+2. Linux/k3s installation and accelerator-specific infrastructure setup.
+3. Basic k3s validation, plus NVIDIA and CUDA validation in NVIDIA mode.
 4. `llm-observability-stack` Helm install.
 5. Observability validation.
 6. Next command and URL hints.
 
+An empty local k3s cluster with only the default k3s system components is a
+valid starting point. `edge install infra` is responsible for preparing the
+NVIDIA runtime and GPU Operator layer before any GPU-backed LLM workload is
+installed.
+
 ## Infra Layer
 
 `edge install infra` uses the local `k3s-nvidia-edge` project as the
@@ -122,6 +153,15 @@ edge uninstall infra --yes
 `edge install observability` uses the local `llm-observability-stack` chart. It
 validates the infra layer first, builds Helm dependencies, installs the chart,
 waits for Ollama, Open WebUI, and OpenTelemetry Collector, then runs validation.
+For GPU profiles, `edge-cli` does not allow bypassing infra validation and
+forcibly keeps `gpu-operator`, `nvidia-device-plugin`, and `dcgm-exporter`
+disabled in the observability chart. Those components belong to
+`k3s-nvidia-edge`.
+
+On single-GPU laptops, run `edge validate infra` before installing
+observability. After Ollama is running and reserving `nvidia.com/gpu: 1`,
+observability validation checks base readiness without launching another CUDA
+pod that would contend for the only GPU.
 
 Useful commands:
 
@@ -178,8 +218,14 @@ commands such as `kubectl`, `helm`, `apt-get`, `systemctl`, `k3s`, and
 ## Documentation
 
 - [Architecture](docs/ARCHITECTURE.md)
+- [Programming language and script boundaries](docs/LANGUAGE-BOUNDARIES.md)
+- [Accelerator selection](docs/ACCELERATOR-MODES.md)
+- [Local dependency and repository inventory](docs/LOCAL-DEPENDENCY-INVENTORY.md)
+- [Live validation - 2026-07-17](docs/LIVE-VALIDATION-2026-07-17.md)
+- [Live validation - 2026-07-08](docs/LIVE-VALIDATION-2026-07-08.md)
 - [Command reference](docs/COMMANDS.md)
 - [Configuration](docs/CONFIGURATION.md)
 - [Troubleshooting](docs/TROUBLESHOOTING.md)
 - [Contributing](CONTRIBUTING.md)
 - [Security](SECURITY.md)
+- [Qwen GGUF runtime evidence companion](https://github.com/Edge-Computing-LLM/qwen-gguf-observability)
diff --git a/docs/ACCELERATOR-MODES.md b/docs/ACCELERATOR-MODES.md
@@ -0,0 +1,43 @@
+# Accelerator selection
+
+`edge install all` and `edge install observability` accept
+`--accelerator auto|nvidia|cpu`.
+
+## Automatic mode
+
+For the full layered installation, `auto` probes the host with `nvidia-smi` before
+making changes:
+
+- NVIDIA detected: install Ubuntu prerequisites, NVIDIA Container Toolkit, k3s,
+  `k3s-nvidia-edge`, and the NVIDIA observability profile.
+- NVIDIA not detected: skip NVIDIA toolkit and GPU Operator components, install or
+  validate basic k3s, and use `values.cpu-k3s.yaml`.
+
+For an observability-only installation, `auto` inspects Kubernetes allocatable
+resources. It selects NVIDIA only when at least one node advertises a positive
+`nvidia.com/gpu` value; otherwise it selects CPU.
+
+## Explicit modes
+
+Use `--accelerator nvidia` where the GPU layer must be present and a missing GPU
+should fail validation. Use `--accelerator cpu` to guarantee that no NVIDIA runtime
+class, GPU request, GPU Operator, device plugin, or DCGM dependency is required by
+the application layer.
+
+An explicit `--profile` on `edge install observability` takes precedence over
+accelerator-based profile selection.
+
+## Detection boundaries
+
+Host detection answers whether NVIDIA infrastructure should be installed. Cluster
+detection answers whether a workload can request a Kubernetes GPU. A host may pass
+`nvidia-smi` while Kubernetes still reports no GPU; that means the GPU layer needs
+installation or repair rather than that a GPU workload is ready.
+
+## Interrupted first installs
+
+If a first Helm client times out while large images continue pulling, Kubernetes
+may later become healthy while Helm remains `pending-install`. With `--yes`, the
+observability installer detects that exact state, reconciles the recorded revision,
+and then proceeds with the normal idempotent upgrade. Other pending Helm states are
+not rewritten automatically.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
@@ -5,14 +5,20 @@
 
 ## Layers
 
-The platform is split into three layers:
+The platform is split into ordered layers:
 
-- `edge-cli`: command-line control plane, configuration, checks, workflows, and
+- Layer 0, `edge-cli`: command-line control plane, configuration, checks, workflows, and
   module orchestration.
-- `k3s-nvidia-edge`: infrastructure layer for Linux, k3s, NVIDIA Container
+- Layer 1, `k3s-nvidia-edge`: infrastructure layer for Linux, k3s, NVIDIA Container
   Toolkit, GPU Operator, RuntimeClass, and CUDA validation.
-- `llm-observability-stack`: LLMOps layer for Helm workloads such as Ollama,
+- Layer 2, `llm-observability-stack`: LLMOps layer for Helm workloads such as Ollama,
   Open WebUI, OpenTelemetry Collector, Prometheus, Grafana, and related tools.
+- Evidence companion, `qwen-gguf-observability`: read-only Qwen runtime contract
+  checks and sanitized point-in-time evidence. It owns no cluster resources and
+  is intentionally outside the ordered install/uninstall layer graph.
+- Future Layer 3 repositories that deploy resources, such as a data/storage
+  stack, should declare their dependency on the previous layer before they can
+  be installed.
 
 ## Module Model
 
@@ -34,6 +40,33 @@ interface:
 - status/logs where relevant
 - safe uninstall behavior
 
+Layer metadata lives in `internal/platform`. Install workflows run in catalog
+order. Uninstall workflows run in reverse order where layers own resources that
+depend on earlier layers.
+
+## Dependency Gates
+
+`llm-observability-stack` GPU profiles require a valid infra layer first:
+
+- `RuntimeClass/nvidia` exists
+- nodes advertise allocatable `nvidia.com/gpu`
+- GPU Operator/device plugin/DCGM are owned by `k3s-nvidia-edge`
+- CUDA validation succeeds from the infra layer
+
+`edge-cli` rejects GPU observability installs that try to skip infra validation.
+It also appends Helm overrides that keep `gpu-operator.enabled`,
+`nvidia-device-plugin.enabled`, and `dcgm-exporter.enabled` false in the
+observability chart.
+
+Full infra validation owns CUDA pod execution. Observability dependency checks
+verify the ready base layer without launching a CUDA pod, so validation remains
+usable after Ollama has reserved the only GPU on a low-VRAM laptop.
+
+After Layer 2 is healthy, `qwen-gguf-observability` may independently read the
+Kubernetes, Helm, Ollama, and `nvidia-smi` status surfaces. This avoids copying
+deployment logic into an evidence repository or coupling evidence capture to an
+`edge install` operation.
+
 ## Execution Model
 
 The CLI is 100% Go. It does not create Bash scripts. Host and cluster operations
@@ -61,5 +94,5 @@ gpu:
   vendor: nvidia
 ```
 
-AMD, Intel, Apple Silicon, and CPU-only accelerator abstractions are out of
+AMD, Intel, Apple Silicon, and CPU-first accelerator abstractions are out of
 scope.
diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md
@@ -44,6 +44,11 @@ edge install all --yes
 `edge install all` executes infra preflight, infra install, infra validation,
 observability install, and observability validation.
 
+For GPU observability profiles, the CLI validates the infra layer first and does
+not allow `llm-observability-stack` to install GPU Operator, NVIDIA device
+plugin, or DCGM exporter. Start from an empty local k3s cluster by running the
+infra install before observability.
+
 ## Validate
 
 ```bash
@@ -57,6 +62,8 @@ Operator pods, GPU allocatable resources, and a CUDA `nvidia-smi` validation pod
 `observability` validation checks infra first, then verifies the Helm release,
 namespace, Ollama, Open WebUI, OpenTelemetry Collector, optional
 Prometheus/Grafana services, pod readiness, and optional Ollama smoke behavior.
+It does not launch a second CUDA pod after Ollama is running, which avoids GPU
+contention on single-GPU workstations.
 
 ## Uninstall
 

diff --git a/docs/LANGUAGE-BOUNDARIES.md b/docs/LANGUAGE-BOUNDARIES.md
@@ -0,0 +1,28 @@
+# Programming language and script boundaries
+
+The organization uses languages by operational responsibility, not by forcing
+every repository into one implementation language.
+
+| Language or format | Use it for | Do not use it for |
+|---|---|---|
+| Go | Durable CLIs, install/uninstall workflows, validation gates, typed configuration, command execution, and cross-repository orchestration | Browser UI or one-off data analysis |
+| Python 3.11 | Structured evidence, benchmarks, API services, notebooks, Kubernetes reporting, and test tooling | Host package installation or a second deployment control plane |
+| Bash | Short transparent wrappers around Helm, kubectl, container image import, and workshop command sequences | Growing business logic, complex parsing, persistent state, or duplicated Go workflows |
+| TypeScript/Vue | Browser dashboards, typed frontend models, chart interaction, and client-side Prometheus parsing | Direct privileged Kubernetes access or host orchestration |
+| Helm/YAML | Declarative Kubernetes resources and deployment profiles | Imperative host installation logic |
+
+## Repository mapping
+
+- `edge-cli`: Go only for product behavior; Make and YAML remain build/CI glue.
+- `k3s-nvidia-edge`: Go for infrastructure workflows and Helm/YAML for the GPU
+  Operator profile.
+- `llm-observability-stack`: Helm/YAML for resources, Go for its legacy/helper
+  CLI, Python 3.11 for application/benchmark/diagnostic code, and Bash only for
+  thin local operator wrappers.
+- `qwen-gguf-observability`: dependency-free Python 3.11 for structured,
+  read-only evidence collection.
+- `Frontend-Edge-LLM-Observability`: Vue 3 and TypeScript for the browser.
+
+New deployable repositories require a clear ownership boundary. A new
+repository should not be created merely to move an existing script to another
+language.
diff --git a/docs/LIVE-VALIDATION-2026-07-08.md b/docs/LIVE-VALIDATION-2026-07-08.md
@@ -0,0 +1,26 @@
+# Live Validation - 2026-07-08
+
+Validated on local Xubuntu 24, single-node k3s, NVIDIA GeForce 940M.
+
+Layer order tested:
+
+1. Empty/mostly-empty k3s baseline: CoreDNS and local-path-provisioner only.
+2. `edge install observability --skip-infra-check --profile geforce-940m-k3s --yes` refused the GPU install.
+3. `edge validate infra` failed before Layer 1 because `nvidia.com/gpu` was not allocatable.
+4. `edge install infra --skip-base-package-install --skip-toolkit-install --skip-k3s-install --yes` installed/upgraded the `k3s-nvidia-edge` GPU Operator wrapper.
+5. `edge validate infra` passed and ran a CUDA `nvidia-smi` pod.
+6. `edge install observability --profile geforce-940m-k3s --yes` deployed Ollama, Open WebUI, Redis, and OpenTelemetry Collector.
+7. Repeated install/validate upgraded the observability Helm release safely.
+8. `edge uninstall observability --yes` removed Layer 2 while Layer 1 stayed healthy.
+9. Reinstall and validation passed again.
+
+Observed ownership:
+
+- `k3s-nvidia-edge` Helm release lives in `gpu-operator`.
+- GPU Operator, NVIDIA device plugin, DCGM exporter, NFD, RuntimeClass, and `nvidia.com/gpu` are provided by Layer 1.
+- `llm-observability-stack` rendered no GPU Operator, NVIDIA device plugin, or DCGM exporter resources.
+- `llm-observability-stack` package did not include the old substrate chart dependencies.
+
+Single-GPU note:
+
+`edge validate infra` runs CUDA validation before Layer 2 is installed. Once Ollama is running and reserving `nvidia.com/gpu: 1`, observability validation checks base readiness without launching another CUDA pod.