Skip to content

from_dbt_models: multi-ref models produce unresolved ref() tables and disconnected column lineage #74

Description

@mingjerli

Summary

At clgraph 0.0.5, Pipeline.from_dbt_models() silently breaks column lineage for any dbt model that references two or more {{ ref('...') }} targets (the standard dbt mart pattern).

Repro (minimal, 3 models)

models/staging/stg_a.sql:  SELECT id, x FROM raw.a
models/staging/stg_c.sql:  SELECT id, k FROM raw.c
models/marts/mart_b.sql:   SELECT a.x, c.k, a.id AS y FROM {{ ref('stg_a') }} a JOIN {{ ref('stg_c') }} c ON a.id = c.id
p = Pipeline.from_dbt_models(project_dir, schema_map={"staging": "staging", "marts": "marts"}, dialect="bigquery")

Observed

  • Table graph contains literal unresolved names "{{ ref('stg_a') }}" / "{{ ref('stg_c') }}" alongside the real staging.stg_a/staging.stg_c tables.
  • marts.mart_b's columns trace to disconnected placeholder nodes (mart_b:unknown.x, …) instead of the real upstream columns — lineage is silently broken, no error or warning.

Expected

All ref() targets resolve to their schema-mapped table names; the mart's columns carry edges from both staging tables' columns.

Root-cause hypothesis

The templated-reference resolution drops every ref() after the first in a query (Jinja-function handling in the template resolver), leaving later refs as raw text through parsing.

Workaround (verified)

Pre-resolve ref() calls to literal target-table names on the output of wrap_dbt_models() before constructing Pipeline directly — clgraph-studio does this in both its demo build script and its live dbt analyze route (mingjerli/clgraph-studio, backend/dbt_io.py::build_dbt_pipeline).

Found while building clgraph-studio's dbt demo; confirmed independently by a second reviewer with the repro above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions