Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ All notable changes to this project are documented in this file.

### Breaking Changes
- **Removed the `TABLASSERT_FULLMAP_SHARDS` environment variable.** The number of on-disk RECORDS shard files is now fixed at the compile-time cap (`16`) and can no longer be overridden at build time; the variable is silently ignored if set. Default builds are unaffected — the previous default was already `16`. The read path still honors the shard count recorded in an existing database's `meta` table (`shards`), so databases built with fewer shards under the old variable continue to open and resolve correctly. See `docs/fullmap.md`.
- **Removed the `build-kg --table-config`/`-tc` flag** (and the `build-kg --fullmap`/`-fm` flag, which existed solely to feed it). `build-kg` now always takes a graph YAML; the throwaway `TEMP_KG` wrapper for building a bare table (Section) config is gone. To build a single table config, wrap it in a graph config — `tablassert agent` already writes a ready `graph.yaml` alongside each build under `builds/<pmc_id>/`. (`validate --schema table` still checks a bare table config on its own.)
- **Renamed `build-kg`'s configuration-file parameter** from `configuration_file` to `graph_configuration_file` to reflect that it is always a graph config. The CLI flags are unchanged (`--configuration-file`/`-f` and positional); only the positional metavar (`GRAPH-CONFIGURATION-FILE`) and the Python / bound-argument name change.
- **`validate` no longer auto-detects the config kind.** The YAML-sniffing heuristic (a top-level `tables` key ⇒ graph, otherwise table) is removed; `validate` now requires an explicit `--schema {graph,table}`/`-s` flag selecting which schema to validate against. `tablassert validate foo.yaml` now fails without `--schema`; use `--schema graph` (validates the `Graph` model and every referenced table) or `--schema table` (validates section syntax only).

## 8.0.1 - 2026-07-30

Expand Down
6 changes: 4 additions & 2 deletions docs/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,10 +214,12 @@ skipped. The `downloads/` payload persists on disk across runs.
### Reusing agent outputs with the full pipeline

The best config's `source.local` points at the downloaded table under `downloads/<pmc_id>/`, so the full
(non-agent) pipeline can reuse the agent's output **without re-fetching**:
(non-agent) pipeline can reuse the agent's output **without re-fetching**. The agent already writes a
ready-to-build `graph.yaml` (wrapping `table.yaml` with the resolved fullmap) into `builds/<pmc_id>/`:

```bash
tablassert build-kg .tablassert/agent/configs/PMC11708054.yaml --table-config --fullmap ./fullmap
cd .tablassert/agent/builds/PMC11708054
tablassert build-kg graph.yaml
```

!!! warning "Not relocatable"
Expand Down
33 changes: 15 additions & 18 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,24 +104,21 @@ Use this to build a KGX NDJSON knowledge graph (nodes, edges, and a Resource Ing
YAML configuration file.

```bash
tablassert build-kg CONFIGURATION-FILE [ARGS]
tablassert build-kg GRAPH-CONFIGURATION-FILE [ARGS]
```

By default the positional `CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is a **graph** YAML.
The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is a **graph** YAML.

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Graph YAML (or a table YAML with `--table-config`) |
| `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Graph YAML |
| `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `biolink:not_significant` edges before resolution) |
| `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact → fuzzy → BioBERT) so low-confidence edges are flagged; requires the `[qc]` extra |
| `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging |
| `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: ≤5 random rows/section, cached to `.head.parquet`, never clobbers a full build |
| `--table-config`, `-tc` | Flag | No | `False` | Build/test one table (Section) config without writing a graph config (wrapped in a throwaway `TEMP_KG` graph) |
| `--fullmap`, `-fm` | Path | No | `./fullmap` | Fullmap path for the throwaway `TEMP_KG` graph when `--table-config` is passed |

```bash
tablassert build-kg graph.yaml --qc --log
tablassert build-kg table-config.yaml --table-config --fullmap ./fullmap
```

Output is written to the current directory as `{name}_{version}.nodes.ndjson`,
Expand All @@ -140,34 +137,34 @@ Output is written to the current directory as `{name}_{version}.nodes.ndjson`,

## validate

Use this to validate a graph or table configuration without running the build — ideal for CI and
pre-commit hooks. Both forms work: `tablassert validate <file>` and `tablassert validate -f <file>`.
Use this to validate a configuration against a schema without running the build — ideal for CI and
pre-commit hooks. The required `--schema` flag selects which schema to validate against (the kind is
no longer sniffed from the YAML).

```bash
tablassert validate CONFIGURATION-FILE
tablassert validate -f CONFIGURATION-FILE
tablassert validate CONFIGURATION-FILE --schema graph
tablassert validate -f CONFIGURATION-FILE --schema table
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

| Option | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Graph **or** table configuration to validate |
| `CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | — | Configuration file to validate |
| `--schema`, `-s` | `graph` \| `table` | Yes | — | Schema to validate against: `graph` validates the `Graph` model **and** every referenced table; `table` validates section syntax only |

The config kind is detected from the YAML: a mapping with a top-level `tables` key is a **graph**
config (validates the `Graph` model **and** every referenced table); anything else is a **table**
config (validates section syntax). Exits non-zero on any schema error. See
[Table Configuration](configuration/table.md) and [Graph Configuration](configuration/graph.md).
Exits non-zero on any schema error. See [Table Configuration](configuration/table.md) and
[Graph Configuration](configuration/graph.md).

```bash
tablassert validate table-config.yaml
tablassert validate graph.yaml
tablassert validate table-config.yaml --schema table
tablassert validate graph.yaml --schema graph
```

---

## Typical workflow

1. Author a table config, then a graph config that references it.
2. `tablassert validate graph.yaml` — fail fast on schema errors.
2. `tablassert validate graph.yaml --schema graph` — fail fast on schema errors.
3. `tablassert build-kg graph.yaml` — produce KGX NDJSON + RIG (add `--qc` to audit mappings).

## Next Steps
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/graph.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Graph Configuration Reference

Graph configurations orchestrate one or more [table configurations](table.md) into a single knowledge-graph build — author one to produce KGX output with `tablassert build-kg`. To build or test a single table config without authoring a graph file, use `build-kg --table-config`, which wraps it in a throwaway graph (see the [CLI reference](../cli.md#build-kg)).
Graph configurations orchestrate one or more [table configurations](table.md) into a single knowledge-graph build — author one to produce KGX output with `tablassert build-kg` (see the [CLI reference](../cli.md#build-kg)). To check a single table config on its own, use `tablassert validate <table.yaml> --schema table`.

## Purpose

Expand Down
73 changes: 20 additions & 53 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,73 +85,44 @@ def _extract_sections_indexed(args: tuple[int, object, Path]) -> tuple[int, list
return idx, to_sections(raw, table) # pyright: ignore


def _load_graph(configuration_file: Path, table_config: bool, fullmap: Path) -> Graph:
def _load_graph(configuration_file: Path) -> Graph:
"""Load and validate the Graph config that drives a build.

By default ``configuration_file`` is a Graph YAML loaded directly. With
``table_config=True`` it is instead a table (Section) YAML wrapped in a
throwaway ``TEMP_KG`` graph so a single table config can be built or tested
without authoring a full graph config; ``contributions`` and ``ui_explanation``
then fall back to the Graph model defaults.

Args:
configuration_file: Graph YAML path, or a table YAML path when ``table_config``.
table_config: When ``True``, wrap the table YAML in a throwaway ``TEMP_KG`` graph.
fullmap: Fullmap path for the wrapped graph when ``table_config`` is ``True``.
configuration_file: Graph YAML path.

Returns:
The validated Graph model.

Raises:
GraphValidationError: If the (possibly wrapped) graph fails Pydantic validation.
GraphValidationError: If the graph fails Pydantic validation.
"""
from tablassert.ingests import from_yaml
from tablassert.models import Graph
from tablassert.progress import flatten_pydantic_error

raw: object
if table_config:
raw = {
"name": "TEMP_KG",
"version": "0.0.0",
"description": "Temporary knowledge graph generated to test a table configuration",
"tables": [configuration_file],
"fullmap": fullmap,
}
else:
raw = from_yaml(configuration_file)
raw: object = from_yaml(configuration_file)
try:
return Graph.model_validate(raw)
except pydantic.ValidationError as e:
raise GraphValidationError(configuration_file, flatten_pydantic_error(e)) from e


def build_pipeline(
configuration_file: Path,
progress: PipelineProgress,
release: bool = False,
qc: bool = False,
log: bool = False,
head: bool = False,
table_config: bool = False,
fullmap: Path = Path("./fullmap"),
configuration_file: Path, progress: PipelineProgress, release: bool = False, qc: bool = False, log: bool = False, head: bool = False
) -> None:
"""Build a knowledge graph from a YAML configuration file.

Runs the six-stage build pipeline: load tables → extract sections → build
Tcodes → collect instructions → build subgraphs → compile graph.

Args:
configuration_file: Path to the graph YAML file (or a table YAML
file when ``table_config`` is ``True``).
configuration_file: Path to the graph YAML file.
progress: Pipeline progress reporter.
release: When ``True``, emit release-mode artifacts.
qc: When ``True``, run quality-control audits on each section.
log: When ``True``, enable per-section verbose logging.
head: When ``True``, preview a random sample of up to 5 rows per section (fast schema/shape check).
table_config: When ``True``, treat ``configuration_file`` as a table
(Section) YAML and wrap it in a throwaway ``TEMP_KG`` graph.
fullmap: Fullmap path used to wrap a table config when ``table_config`` is ``True``.

Raises:
GraphValidationError: If the graph YAML fails Pydantic validation.
Expand All @@ -164,7 +135,7 @@ def build_pipeline(

# Stage 1/6: load tables.
progress.stage("Loading Tables")
g: Graph = _load_graph(configuration_file, table_config, fullmap)
g: Graph = _load_graph(configuration_file)
# imap_unordered yields in completion order, so each worker carries its input
# index and we reassemble by index to keep raw[i] aligned with g.tables[i].
start, advance, _ = progress.section_loop(len(g.tables), "Load")
Expand Down Expand Up @@ -492,36 +463,32 @@ def _download_detail(downloaded: int, total: int) -> str:

@APP.command(name="build-kg")
def build_kg(
configuration_file: Annotated[Path, cyclopts.Parameter(name=["--configuration-file", "-f"])],
graph_configuration_file: Annotated[Path, cyclopts.Parameter(name=["--configuration-file", "-f"])],
release: Annotated[bool, cyclopts.Parameter(name=["--release", "-r"], negative="")] = False,
qc: Annotated[bool, cyclopts.Parameter(name=["--qc", "-q"], negative="")] = False,
log: Annotated[bool, cyclopts.Parameter(name=["--log", "-l"], negative="")] = False,
head: Annotated[bool, cyclopts.Parameter(name=["--head", "-hd"], negative="")] = False,
table_config: Annotated[bool, cyclopts.Parameter(name=["--table-config", "-tc"], negative="")] = False,
fullmap: Annotated[Path, cyclopts.Parameter(name=["--fullmap", "-fm"])] = Path("./fullmap"),
) -> None:
"""Build a knowledge graph from a YAML configuration file.

By default the positional config is a Graph YAML. With ``--table-config`` it is a
table (Section) YAML wrapped in a throwaway ``TEMP_KG`` graph (``--fullmap`` sets
the fullmap path) so a single table config can be built or tested without
authoring a full graph config.
The positional config is a Graph YAML that orchestrates one or more table
configs into a single knowledge-graph build.
"""
run(6, build_pipeline, configuration_file, release=release, qc=qc, log=log, head=head, table_config=table_config, fullmap=fullmap)
run(6, build_pipeline, graph_configuration_file, release=release, qc=qc, log=log, head=head)


@APP.command(name="validate")
def validate(configuration_file: Annotated[Path, cyclopts.Parameter(name=["--configuration-file", "-f"])]) -> None:
"""Validate a graph or table YAML configuration file.
def validate(
configuration_file: Annotated[Path, cyclopts.Parameter(name=["--configuration-file", "-f"])],
schema: Annotated[Literal["graph", "table"], cyclopts.Parameter(name=["--schema", "-s"])],
) -> None:
"""Validate a YAML configuration file against the graph or table config schema.

Detects the config kind from the YAML: a mapping with a top-level ``tables`` key
is a graph config (validates the Graph model AND every referenced table); anything
else is treated as a table config (validates section syntax only).
``--schema graph`` validates the Graph model AND every referenced table; ``--schema
table`` validates section syntax only. The schema is selected explicitly rather than
sniffed from the YAML, so a config is always checked against the schema you expected.
"""
from tablassert.ingests import from_yaml

loaded: object = from_yaml(configuration_file)
if isinstance(loaded, dict) and "tables" in loaded:
if schema == "graph":
run(2, validate_graph_pipeline, configuration_file)
else:
run(3, validate_pipeline, configuration_file)
Expand Down
11 changes: 6 additions & 5 deletions tests/test_agent_storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -358,11 +358,12 @@ def fake_fetch(pmc_id: str, outdir: Path, *, timeout: int = 120) -> list[Path]:
def test_supervisor_best_config_pipeline_reuse(tmp_path: Path, fullmap_db: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The BEST config references the STABLE download and rebuilds from a FRESH cwd (REQ-LAYOUT-5/8).

Why: the pipeline-reuse contract. ``tablassert build-kg <cfg> --table-config --fullmap <fm>`` must
be able to reuse the supervisor's accepted config WITHOUT re-fetching: its ``source.local`` must be
the REAL, persisted download under ``state_dir/downloads/<pmc>/`` (not a temp path), and because that
path is ABSOLUTE the config must build from ANY cwd. This proves the download is real + referenced
and that the best config is self-sufficient for downstream reuse.
Why: the pipeline-reuse contract. The supervisor's accepted (best) config must be reusable WITHOUT
re-fetching by passing it straight back through the Python API (``build_and_audit`` — the same call
the supervisor uses): its ``source.local`` must be the REAL, persisted download under
``state_dir/downloads/<pmc>/`` (not a temp path), and because that path is ABSOLUTE the config must
build from ANY cwd. This proves the download is real + referenced and that the best config is
self-sufficient for downstream reuse.
"""
pytest.importorskip("smolagents")
import yaml
Expand Down
Loading
Loading