Feature: Add consumption and observability to dbt Artifacts

[mtcarlone]

Add consumption and run-health marts (Snowflake)

Background

dbt_artifacts already captures everything needed to answer two questions teams ask constantly, but it ships no models that answer either:

"How much of our dbt platform allowance have we burned, and when do we blow through it?" dbt bills on Successful Models Built — every model built in a deployment environment via orchestration, where views, tables and incrementals count equally and reruns count again. That usage is visible in the dbt platform UI, but not in the warehouse alongside every other cost metric a team tracks, and it is not projected forward. The package has recorded every model execution and invocation since day one; the meter is a select away.

"Why is the nightly job slower, and which tests do we just retry until they pass?" Runtime, status and timing are already captured per node execution, but every team that wants regression detection, flake identification or a run-health dashboard builds the same SQL by hand.

Both are pure analytics over existing source tables. This PR adds those marts. No new upload datasets, no changes to the on-run-end hook, and no changes to any existing model — the column-order contract between the upload macros, source models and column-name lists is untouched.

Scope is deliberately Snowflake-only for this first cut. Every new model is gated with config(enabled = target.type == "snowflake") and the one helper without portable SQL degrades safely (see below), so consumers on the other six adapters see no behavioural change at all. Cross-adapter implementations are a follow-up.

Key Changes

Consumption marts

• fct_dbt__consumption_daily (+ _detail) — one row per UTC day × meter. Two meters ship: smb (Successful Models Built, encoding dbt's published rules — success status, deployment invocations only, run/build/retry commands, all materializations counted equally, reruns counted again) and active_target_tables (distinct models, seeds, snapshots and tests active per UTC day — the upper bound of what a table-day meter would count). Meters are rows rather than columns, so additional meters are additive. The daily model is a strict roll-up of the detail model, so the two grains always reconcile.
• fct_dbt__consumption_forecast — one row per billing month × meter: month-to-date quantity, plan allowance, weekday-aware run rate, projected month-end total, projected allowance-breach date, and percent of allowance consumed. The projection averages per ISO day-of-week over a trailing window (including zero-days) rather than a straight mean, so weekly seasonality does not distort it. forecast_exceeded_date reports the real crossing day in both directions: for elapsed months it finds the first day the actual running total reached the allowance, rather than collapsing to month-end.
• fct_dbt__consumption_by_model — one row per billing month × model: SMB burn, share of monthly total, build cadence, distinct days built, total runtime, and an estimated cost of reusing rather than rebuilding, for teams evaluating state-aware builds.

Observability marts

• fct_dbt__run_health_daily (+ _detail) — per-day invocation counts, node successes/failures/errors/skips across models, seeds, snapshots and tests, success rate, runtime totals, and failed-invocation counts. Unlike the consumption marts this covers all invocations, not just deployment ones.
• fct_dbt__model_performance — per day × model runtime statistics against a trailing weekday-aware baseline, with a regression ratio and an is_regressed flag. Full-refresh executions are excluded from both current-day stats and the baseline (a full refresh is not a regression) and surfaced as a separate count, and the flag requires a minimum baseline sample so new models do not fire spuriously.
• fct_dbt__flaky_tests (+ _detail) — tests that flip from failing to passing with no successful rebuild of the attached model in between, rolled up monthly with a flake rate. Where parentage cannot be resolved the flip is still reported with the uncertainty exposed as a column rather than silently dropped.
• fct_dbt__dag_bottlenecks (+ _detail) — measured stall time: within an invocation, how long each model waited on the latest-completing parent that gated it, attributed to that binding parent. This is observed wall-clock gating from execution timestamps, not graph estimation. Descendant counts and blocking scores are intentionally out of scope here.
• dim_dbt__lineage_edges — one row per parent → child edge, derived from the depends_on_nodes arrays already captured for models, snapshots and tests. Turns lineage that was only queryable as a nested array into a joinable table.

Supporting macros

• cast_to_utc_date(), median() / percentile() / p95(), and flatten_json_array() in macros/database_specific_helpers/, all dispatched in the package's existing pattern.
• classify_invocation_billing() — resolves an invocation to deployment or development by documented precedence (dbt Cloud job id present, then configurable target names, then an optional environment-variable marker), since only deployment builds are billed.
• get_smb_allowance() — resolves the monthly allowance from an explicit override, else a plan-name default, else none.
• is_between() generic test for range assertions on the new models.

Configuration

All new behaviour is var-driven with working defaults, so the package behaves identically for anyone who sets nothing: dbt_artifacts_billing_plan, dbt_artifacts_smb_allowance, dbt_artifacts_deployment_targets, dbt_artifacts_deployment_env_var, dbt_artifacts_count_all_invocations, dbt_artifacts_datt_price, dbt_artifacts_run_rate_days, dbt_artifacts_regression_threshold, dbt_artifacts_regression_min_samples, dbt_artifacts_flaky_min_flips.

Backwards compatibility

• Additive only: no existing model, macro, source table or column was modified, renamed or removed.
• New models are Snowflake-gated, so they are absent from the graph on other adapters.
• flatten_json_array() has no portable implementation yet. Its default emits a parse-safe placeholder when execute is false, so dbt parse and dbt compile succeed unchanged on every adapter, and raises a clear compiler error only if a model using it is actually built on an unsupported adapter — which the gating already prevents.
• Every new var has a default; nothing new is required.

Tests

Eight singular tests in the integration project cover both mart families: SMB rows present and positive, forecast returning exactly one row per meter for the current billing month with a resolved allowance, by-model totals reconciling to the daily grain, lineage edges covering every model with dependencies, and validity/range assertions on the performance, flaky-test and bottleneck marts. The integration project sets a billing plan and deployment-target list so the consumption marts have classified invocations to aggregate; those vars are inert on non-Snowflake targets.

Not included

Package version bump and README documentation for the new models and vars. These marts are new views, so consumers will need to re-run dbt run --select dbt_artifacts after upgrading — the release should be a minor version, and the notes should carry that instruction.