Feature: Run FinOps hubs on your own hardware - #2187
Merged
Conversation
Local install of FinOps Hubs: runs the toolkit's ADX KQL against a local
Kusto Emulator (Kustainer) with the real two-database topology
(Ingestion + Hub), so dashboards / Power BI / the query catalog run
unchanged against a local endpoint.
T-5000.1 — load-ftk-kql.ps1 creates the Ingestion + Hub databases and loads
them by combining the toolkit's own finops-hub/.build.config fabric-setup
bundles (real v1_0 included; database('Ingestion'). cross-DB references
preserved; zero object renames — strict name parity). No hand-coded load
order, no strip hack, no v1_0 stubs.
T-5000.2 — ingest.ps1 -> Ingestion; ftk.ps1 -> Hub; run-parity-checks.ps1 is
DB-aware (raw/final checks on Ingestion, Hub-view checks 9/10 on Hub) and
adds a name-parity check. Verified end-to-end on R&D data: Costs and Prices
ingested at exact parity (BilledCost preserved to $0.00; Prices raw==final
12,735,587), Hub cross-DB queries resolve, parity suite 10 pass / 0 fail /
2 manual.
Prototype docs/notes are intentionally not migrated here yet — they are
authored cleanly (sanitized, two-DB-accurate) in the docs task (T-5000.7).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Red-team caught that `make kql` defaulted to Ingestion, so Hub view-function queries (e.g. Costs_v1_2) failed with 400 unless DB=Hub was passed. The DB var was shared with the parity recipe, which needs Ingestion as its base DB. Split into KQL_DB (?= Hub, used by `make kql`) and PARITY_DB (?= Ingestion, passed to run-parity-checks.ps1). ingest-status now passes KQL_DB=Ingestion explicitly since it inspects Ingest_Manifest in Ingestion. Header/help/topology comments updated. ingest unaffected (ingest.ps1 defaults to Ingestion). Verified: `make kql QUERY='Costs_v1_2 | count'` -> 1350561 (Hub) with no override; KQL_DB=Ingestion override works; make parity still 10/0/2 + name parity PASS; make ingest-status reads Ingest_Manifest on Ingestion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…00.4)
Document the data-in path: Cost Management export configuration via the
toolkit's New-/Get-/Start-FinOpsCostExport cmdlets (Azure-only; ftklocal
consumes the produced FOCUS parquet), and the local staging contract that
ingest.ps1 consumes — export/{scope}/{type}/{period}/{run-uuid}/ with
manifest.json, the ms--focus-cost / ms--pricesheet dataset tokens, latest-run
selection, SHA-256 idempotency, and supersede semantics (correct/replace data
by staging a NEW run-uuid; same run-uuid + changed file double-ingests).
Maps the msexports/ (no hub) and ingestion/ (hub deployed) source modes;
auto-download bridge deferred (T-5000.9). Contract cross-checked against
ingest.ps1.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on (T-5000.10) Add a sanitized settings.example.json (HubInstance schema: scopes + retention) and scripts/settings.ps1 (init/show/get) for local-hub fidelity; gitignore the real settings.json. Wire load-ftk-kql.ps1 to derive the raw soft-delete window ($$rawRetentionInDays$$) from settings.json retention.raw.days with a 3650 fallback and a guard (0/negative/non-numeric -> fallback) plus a warning when a positive value is surprisingly short (<90 days) for local analysis. Document the config-container decision: Mode A (ingestion container) needs no config; Mode B (msexports) would need the msexports->ingestion conversion + the in-repo Exports/schemas (deferred to the bridge, T-5000.9). The ADX/KQL layer consumes no config schemas; load-ftk-kql.ps1 reads settings.json only for retention. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Localize the FinOps hub ADX dashboard for the local Kusto emulator: point the single data source at http://localhost:8082 / database Hub (135 tiles, 87 queries unchanged from upstream). Add notes/dashboard.md documenting the proven ftk.ps1 CLI path, the ADX web UI as best-effort with the HTTPS->HTTP mixed-content caveat, and an ngrok HTTPS-tunnel workaround. Power BI dropped (local Mac, no AAD endpoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e-ingest guard (T-5000.3) Add proactive per-extent chunked backfill for large final tables (default threshold 2,000,000 rows OR >4 source extents, both env-configurable via BACKFILL_CHUNK_ROW_THRESHOLD / BACKFILL_CHUNK_EXTENT_THRESHOLD), so the Prices transform never attempts a single-pass backfill that OOMs; Costs stays single-pass with the reactive chunked fallback intact. Add a fail-fast guard: re-staging a file under the same (scope,type,period,run-uuid) with a changed checksum now halts with a non-zero exit and a clear message instead of silently appending duplicate rows. Harden Invoke-KustoPost strict-mode catch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sers/ literal) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…resolution (T-5000.8) Add the finops-hub-local agent skill (SKILL.md thin router, README, local-vs-remote reference) that drives the local Kusto-emulator FinOps hub via the ftk.ps1 CLI and the finops-hub-local Makefile, reusing the published query catalog (no new KQL). Also fix ftk.ps1 catalog discovery for the in-repo layout: Get-FtkRepo assumed the old prototype's sibling-folder layout (<parent>/finops-toolkit) and failed to find src/queries/catalog when ftklocal lives inside the repo, breaking 'ftk list' and 'ftk run'. Resolve the toolkit root in-repo (three levels up from the script dir) with a git-toplevel fallback and the legacy sibling layout as last resort. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Under Set-StrictMode + $ErrorActionPreference='Stop', calling '& git' when git is absent throws CommandNotFoundException and skips the legacy sibling-layout fallback. Guard the git-toplevel probe with Get-Command so a missing git degrades gracefully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…alog fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…+ harden Prices value parity (T-5000.3) Per architect-duo ruling: drop the extents>4 proactive-chunk criterion (OOM is row-driven; >4 extents mis-fires on every multi-part export and would chunk Costs, which single-passes safely at 15 extents). Gate proactive chunking on rawCount > 2,000,000 alone; keep the reactive single-pass->chunked fallback. Extent count is now informational logging only. Remove the BACKFILL_CHUNK_EXTENT_THRESHOLD knob; note that adding a new RawToFinalPolicy table needs a fresh single-pass OOM measurement. Fix the Makefile threshold comment (3,000,000 -> 2,000,000). Harden parity CHECK 13 (Prices value parity): emit delta_pct so the runner enforces a 0.1% relative tolerance, with a sentinel that fails on zero rows or row-count mismatch so the check cannot pass vacuously. Verified: real data 9,804,381 rows / sums equal at $58.39B -> PASS; x1.01 corruption -> FAIL; zero rows -> FAIL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…uble-ingest guard, Prices value parity) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ss-db strip in ftk.ps1 (T-5000.7)
Add user-facing docs under src/templates/finops-hub-local/: README.md (quickstart,
two-database Ingestion+Hub architecture, ingest with the fail-fast double-ingest guard
and automatic chunked backfill, querying via ftk.ps1 + the agent skill, dashboard
best-effort, measured limits table, troubleshooting), notes/ftk-kql-adaptations.md (the
load-time vs ftk.ps1 on-read adapter layers), notes/parity-gaps.md (the 13-check parity
suite, grounded in a live run).
Also remove the now-stale database('Ingestion'). strip from the ftk.ps1 on-read adapter:
on the two-database topology cross-database references resolve correctly, so stripping
them would break any catalog query that legitimately references an Ingestion table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…trip removal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add notes/performance.md with measured memory characterization on MEM_LIMIT=16g (amd64-on-Rosetta), instrumented via cgroup v2: cold loaded floor ~1.12 GiB (7%), hot post-ingest working set 13.99 GiB (87.4%), Prices chunked-backfill peak 15.03 GiB (94%), and the single-pass Prices transform ceiling where Kusto self-terminates at 15.12 GiB (~94.5%). Includes the row-count to memory curve, the ingest wall-clock (raw 7.0 min, full 14.0 min), and an operating-states table that answers when the engine approaches the limit (only during ingest+transform of a 28M-row dataset). Reconcile the README limits table with these measured numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…red) Replace the speculative 'mixed-content blocked / not proven' caveat with measured findings: the Kusto emulator returns CORS headers for https://dataexplorer.azure.com (OPTIONS 204 + POST 200 with Access-Control-Allow-Origin), real dashboard tile queries return renderable data over that path, and dashboard.json (schema v60, 135 tiles / 87 queries / 12 pages) has zero dangling references so it imports cleanly. localhost is exempt from mixed-content blocking; the real browser gate is the Edge 143+/Chrome 142+ Local Network Access permission prompt (click Allow), with browser-flag and ngrok fallbacks documented for locked-down environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… + cross-platform bring-up Add per-OS install guides (notes/install-windows.md primary, notes/install-mac.md secondary) and make the README quickstart cross-platform: 'docker compose up -d --wait' (honors the container healthcheck) plus the self-contained 'pwsh scripts/*.ps1' commands replace the Unix-only 'make' front door, which remains a macOS/Linux convenience. The scripts already self-resolve HOST_PORT from .env, so they run standalone on Windows. Windows guide covers Docker Desktop + WSL 2, PowerShell 7 (not Windows PowerShell 5.1), WSL 2 memory tuning, forward-slash EXPORT_DIR, and Windows-specific troubleshooting; notes that linux/amd64 runs natively on Windows x64 (no emulation) so it outperforms the Apple-Silicon numbers in performance.md. Supporting correctness: add .gitattributes (LF for .ps1/.sh/.kql/.json/Makefile) to prevent CRLF corruption on Windows checkout, and '#Requires -Version 7.0' to the entry scripts so PowerShell 5.1 fails with a clear message instead of cryptic parser errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run repo Prettier (.prettierrc) over the finops-hub-local markdown so the docs conform to the toolkit coding standard (docs-wiki/Coding-guidelines.md): aligned table columns and underscore emphasis. Formatting only — no content, measured numbers, or links changed. 'prettier --check' now passes for all finops-hub-local markdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Matches the repo-wide markdown convention (all .md stored LF) and prevents CRLF drift for Windows contributors under core.autocrlf=true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add notes/build-by-hand.md — the manual runbook for standing up a local FinOps hub: six steps, about a dozen raw commands (start emulator, create the two databases, load the schema, ingest Parquet, build the final tables, query). Each schema bundle is a single .execute database script command, so loading is one POST per database; the doc shows the exact curl/KQL for every step, verified against the emulator (118/118 statements load in one call). A closing table maps each manual step to the helper that automates it, so the scripts are no longer a black box. Link it from the README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The engine runs a whole bundle natively via '.execute database script with (ContinueOnErrors=true)', returning a per-statement result table. So wrap each built bundle and POST it once instead of splitting it into ~157 commands and posting each in a loop. Removes the command splitter and per-command loop (~48 lines); per-statement diagnostics are preserved by parsing the result table. Verified: 118/118 Ingestion and 39/39 Hub statements load in one call each, schema reload is non-destructive (data intact at the known total), and the dry-run path is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
All feedback from Roland Krummenacher (@RolandKrummenacher)'s review has been addressed in commit b4d5eb2 (and a subsequent merge from dev + CI fix):
|
Contributor
|
Hubs (manual) deployed to |
Michael Flanakin (flanakin)
left a comment
Michael Flanakin (flanakin)
left a comment
Collaborator
There was a problem hiding this comment.
🤖 [AI][Claude Code] PR Review
Summary: A well-executed feature — verified end-to-end against a real Kusto emulator, with clean unit tests (33/33 passing locally), zero PSScriptAnalyzer warnings, and a KQL open-data script whose schema I cross-checked against both the deployed hub's raw table definitions and the Data Factory pipeline commands in app.bicep — they match exactly. I also ran the actual toolkit build and confirmed it produces the new finops-hub-local-opendata.kql release artifact correctly. No blockers found.
⚠️ Should fix (1)
-TimeoutSecis missing from theInitialize-FinOpsHubLocaldocs-mslearn page.
💡 Suggestions (3)
- A selective
-Confirmdecline could hit a null-reference error instead of a friendly message. - The new local-hub open-data KQL duplicates
app.bicep's inline pipeline commands with no automated check keeping them in sync. - The new guide isn't cross-linked from
finops-hubs-overview.mdordeploy.md.
- docs: add missing -TimeoutSec to Initialize-FinOpsHubLocal reference (should-fix) - Initialize-FinOpsHubLocal.ps1: guard against a selective -Confirm decline leaving \ partially populated, which previously threw a raw null-reference exception instead of a friendly, localized error - docs: cross-link run-hubs-locally.md from finops-hubs-overview.md and deploy.md so it's discoverable outside of TOC.yml Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 2, 2026 09:46 — with 
GitHub Actions
Error
GitHub Actions
Error
Update ms.date in docs-mslearn/toolkit/changelog.md, hubs/run-hubs-locally.md, and powershell/powershell-commands.md, which stayed at 07/31/2026 while still carrying body content changes vs the PR base. The 'Update ms.date in changed markdown files' check requires ms.date to match today's date whenever body content changed vs base and the bot's own auto-fix push is blocked by branch protection (GH006), so this is applied manually per repo convention. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Hubs (manual) deployment failed. View logs. |
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 2, 2026 09:51 — with
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
|
Hubs (manual) deployment failed. View logs. |
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 2, 2026 09:54 — with
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
|
Hubs (manual) deployment failed. View logs. |
Brett Wilson (MSBrett)
temporarily deployed
to
ftk-pr
August 2, 2026 09:59 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
|
Hubs (manual) deployed to |
Roland Krummenacher (RolandKrummenacher)
approved these changes
Aug 3, 2026
Michael Flanakin (flanakin)
approved these changes
Aug 12, 2026
Michael Flanakin (flanakin)
added
Status: ▶️ Ready
# Conflicts: # docs-mslearn/toolkit/changelog.md
Contributor
|
Hubs (manual) deployed to |
fix(tests): accept SHA-pinned action refs in update-mslearn-dates workflow test PR #2233 pinned GitHub Actions to full-length commit SHAs on dev, which made this test's checkout/changed-files regexes (expecting @v\d+ tags) stale. Widen them to accept either a 40-char SHA or a version tag. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
|
Hubs (manual) deployed to |
# Conflicts: # src/powershell/Tests/Unit/Action.UpdateMsLearnDates.Tests.ps1
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 13, 2026 07:38 — with
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
|
Hubs (manual) deployment failed. View logs. |
Contributor
|
Test environments cleaned up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠️ Description
Adds and UAT-hardens a self-hosted path for FinOps hubs: a documented way to run the hub analytics layer on your own hardware using the Kusto emulator. Customers, consultants, contributors, and agents can exercise the same hub KQL, schema, transforms, update policies, and open-data enrichment without deploying the full Azure-hosted stack.
This PR is no longer docs-only. Based on review feedback (see below), it now also ships the supporting automation so the guide stays correct and easy to follow:
finops-hub-local-opendata.kqlbuild artifact — the open-data load script is generated by the toolkit build from the same source a deployed hub uses (.build.configcombineKql), so the guide downloads and runs it instead of hand-codingexternaldataschemas that could silently drift.Initialize-FinOpsHubLocalPowerShell command — sets up a local hub against a running emulator in one call: creates theIngestionandHubdatabases, downloads the released setup scripts, applies raw retention, and loads open data with verify-and-retry. It does not install or manage Docker, and it does not ingest cost data.Prices_final_v1_2.Why this changed from the original docs-only PR
Michael Flanakin (@flanakin) requested removing the agent skill (anti-pattern) and asked whether the fragile inline blocks could be build-generated and driven by a PowerShell command. This PR does exactly that: the skill is removed, the open-data load is build-generated, and the deterministic steps are a tested command — no Docker management in the module.
Verification
Verified end-to-end against a real Kusto emulator (Docker
kustainer-linux) with real release artifacts served over HTTP and real FOCUS data (Cost Management export, billing account 8611537, 2025-04). No mocks.src/powershell/Tests/Integration/Initialize-FinOpsHubLocal.Tests.ps1: 7/7 passed.Costs_raw117,559 →Costs_final_v1_2117,559;Prices_raw/Prices_final_v1_2853,636; prices present before the costs phase;Costs_rawretention90.00:00:00; cost enrichment 117,559/117,559.externaldataread after setup can silently return 0 rows; the command and the guide now load, verify, and retry.Boundaries
The Kusto emulator is not for production: no security, no managed ingestion, limited by host capacity. The local hub is an analysis and validation tier, not a replacement for an Azure Data Explorer or Microsoft Fabric hub.
No linked issue.
📷 Screenshots
Not applicable.
📋 Checklist
🔬 How did you test this change?
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?📖 Did you update documentation?