[FEATURE](pyspark) PySpark validation generated from the Pydantic schema

[Seth Fitzsimmons (sethfitz)]

Summary

Adds a new runtime package (overture-schema-pyspark) plus a new output target in overture-schema-codegen that emits PySpark validation expressions and conformance tests from the same Pydantic models that define the schema. Ships an overture-validate CLI (to be merged with the overture-schema CLI later; that will be an opportunity to improve its ergonomics) for running the validation against Parquet on disk or in S3.

PySpark plugs in as a peer of the existing Markdown output target: same ModelSpec extraction, same four-layer architecture (Discovery -> Extraction -> Output Layout -> Rendering), new pipeline module. The codegen abstraction is model-general (ModelSpec = RecordSpec | UnionSpec: one record that compiles to a Spark StructType, or a tagged union of records); in this schema every model is an Overture feature, so the generated modules are organized by theme/feature. See packages/overture-schema-codegen/docs/design.md for the full picture; the "PySpark Pipeline" section there covers the new stages in detail.

Supersedes the draft #518: generated code is dropped from git history (regenerated on demand), the review-facilitation fixups are folded into the main commit, and the branch is rebased on main.

What's in the PR

packages/overture-schema-pyspark/ -- runtime. Public API in validate.py (validate_model, explain_errors), schema comparison in schema_check.py, dataclasses in check.py, the overture-validate CLI in cli.py, and shared expression building blocks in expressions/{constraint_expressions,column_patterns,_schema_structs}.py. The per-feature expression modules under expressions/generated/overture/schema/<theme>/<feature>.py and per-feature conformance tests under tests/generated/overture/schema/<theme>/test_<feature>.py are emitted by codegen into a gitignored generated/ boundary that make generate-pyspark wipes and recreates -- they are not committed to the repository. _registry.py walks that tree at import time and exposes REGISTRY: dict[str, ModelValidation] keyed by each module's ENTRY_POINT value (e.g. "overture.schema.transportation:Segment"), reading each module's emitted MODEL_VALIDATION constant, plus a parallel PARTITION_MAP derived from each module's PARTITIONS dict for partitioned features.

Validation degrades gracefully when columns are skipped (--skip-columns / --suppress) or structurally absent from the data. Every check declares the top-level columns it reads, and a check over a missing column -- top-level, nested inside an array or map (sources[].confidence resolves to sources), or model-level -- is dropped before Spark planning rather than raising a raw AnalysisException. ValidationResult.absent_columns surfaces the columns dropped for this reason, and the CLI backstop classifies any residual planning error through the structured PySpark error API so a generator bug propagates as a traceback instead of being mistaken for a missing column.

This has been tested to work with Spark versions 3.4.0 - 4.1.1 (and the lowest-direct CI check will verify that it continues to work with the lowest declared PySpark version, which is currently 3.4).

Adam Lastowka (@Rachmanin0xFF) the public API changed since the previous version you tested (the entry point is now validate_model, returning a ValidationResult, in an attempt to simplify how it gets integrated with the CDP). If you point me to its current state, I can propose an update that uses the new API.

packages/overture-schema-codegen/src/overture/schema/codegen/pyspark/ -- new output target. Pipeline stages:

ModelSpec (from extraction)
    |
constraint_dispatch.py    constraints -> ExpressionDescriptor / ModelConstraintDescriptor
    |
check_builder.py          FieldShape tree -> Check / ModelCheck IR
                          (check_ir.py; resolves array nesting, map key/value
                          projection, variant gating via
                          FieldPath = ScalarPath | ArrayPath | MapPath and Guard
                          sum types)
schema_builder.py         FieldShape tree -> SchemaField list (StructType source)
test_data/                ModelSpec -> BASE_ROW_SPARSE / BASE_ROW_POPULATED, scaffold,
                          invalid_value
    |
renderer.py               Check IR -> per-feature expression module
test_renderer.py          Check IR -> per-feature conformance test module
    |
pipeline.py               orchestrates, returns GeneratedModule list

make generate-pyspark wipes both generated/ trees and recreates them; make check (and make test-all) regenerate them and run the generated conformance tests. The trees are not tracked in git.

What's covered

Every constraint Pydantic enforces today is dispatched to a PySpark expression, which allows Spark to push down predicates and take advantage of Catalyst to avoid unnecessary deserialization:

• Field constraints: Ge / Gt / Le / Lt / Interval, ArrayMinLen / ArrayMaxLen / ScalarMinLen / ScalarMaxLen (typed length constraints split by attachment layer), StrippedConstraint, PatternConstraint, UniqueItemsConstraint, GeometryTypeConstraint, JsonPointerConstraint.
• Map key/value constraints: dict-typed fields (e.g. names.common, a dict[LanguageTag, StrippedString] present on most named features) have their per-key and per-value constraints validated by projecting map_keys / map_values and applying the same field checks. A map value that is a sub-model is descended into, validating its field and model-level constraints. dict[K, Any] maps (e.g. Infrastructure.source_tags) carry no constraint and emit no checks; container-valued map values (a list or map inside a map value) are not yet supported and raise at generation time rather than emit an unanchored check.
• NewType overrides: LinearlyReferencedRange (length / bounds / order).
• Base-type overrides: HttpUrl (format + length), EmailStr, BBox (completeness -- at parity with Pydantic; coordinate ordering and range validation is deferred to Define validation policy for antimeridian-crossing bboxes and geometries #531).
• Model constraints: RequireAnyOfConstraint, RadioGroupConstraint, RequireIfConstraint, ForbidIfConstraint, MinFieldsSetConstraint. MinFieldsSetConstraint mirrors Pydantic's model_fields_set semantics: required fields are always set (the constructor enforces them) and count toward the threshold alongside any explicitly-set optional fields. NoExtraFieldsConstraint is intentionally skipped, as its behavior is implicit when using DataFrames (extra top-level columns are caught by the schema comparison before per-row validation).

Known semantic gaps / differences

Divergences from Pydantic:

• UniqueItemsConstraint uses Spark's array_distinct, which compares whole elements with structural equality on raw stored values. Pydantic compares normalized Python objects -- e.g., list[HttpUrl] is compared after URL normalization (such as ensuring that URLs like http://example.com contain a trailing /: http://example.com/). The PySpark check catches exact duplicates, not duplicates after normalization.
• require_any_of checks isNotNull as a proxy for Pydantic's model_fields_set. Parquet has no equivalent of "explicitly provided" because of the tabular nature of DataFrames; isNotNull is stricter (it rejects fields explicitly set to null). This matches the current Pydantic behavior, which also rejects fields explicitly set to None.
• Regex checks run on the JVM, so a PatternConstraint is evaluated with Java's regex dialect rather than Python's -- the two agree for the schema's patterns, but ASCII-vs-Unicode shorthand classes (\w, \d) and interior-newline handling differ in general.

A general code-generation gap is tracked separately in #568: collection types set, variable-length tuple[T, ...], and Sequence are mis-handled by the extractor for both the Markdown and PySpark targets. No Overture model uses these types today.

CLI

$ overture-validate --help
Usage: overture-validate [OPTIONS] FEATURE_TYPE PATH

  Validate Overture data at PATH and write annotated Parquet.

Options:
  -o, --output TEXT            Output path for validated Parquet.
  --head INTEGER               Error rows to display.  [default: 20]
  --conf TEXT                  Spark config key=value pairs.
  --count-only                 Report error count only; skip explain/unpivot.
  --skip-schema-check          Warn on schema mismatches instead of aborting.
  --skip-columns TEXT          Columns declared absent from data; skips their
                               checks.
  --ignore-extra-columns TEXT  Extra data columns to ignore in schema
                               comparison.
  --suppress TEXT              Suppress checks: FIELD (all checks) or
                               FIELD:CHECK (specific).
  --help                       Show this message and exit.

Output is one row per violation: feature ID, theme/type, failing field, check name, message, offending value.

# Local Parquet
overture-validate segment samples/segment.parquet --count-only

# Real release prefix (expect bbox Float-vs-Double mismatch -> use --skip-schema-check)
# partitions (theme=places, type=place) will automatically be detected
# note that the s3a protocol must be used when AWS-provided Hadoop filesystem JARs are not available
overture-validate place s3a://overturemaps-us-west-2/release/<release>/ --skip-schema-check --head 50

Testing

overture-schema-codegen generates conformance tests alongside the PySpark expressions as a sanity-check by evaluating expected-good and expected-bad rows within a single DataFrame (setup/teardown overhead for DataFrame-per-test-case was much too high). Each feature gets two baseline rows -- BASE_ROW_SPARSE (required fields only) and BASE_ROW_POPULATED (every optional field filled in) -- and every scenario is exercised against both, so checks that depend on optional structure being present are covered as well as the minimal-row case.

make check generates tests and runs the full suite over everything, which use PySpark and increase test runtime (they run in just over a minute). testmon was introduced to improve the developer experience by skipping tests that weren't affected by recent edits.

Notes for review

• Generated code is not committed. The expressions/generated/ and tests/generated/ trees are produced by make generate-pyspark and gitignored; make check and make test-all regenerate them before running. The review surface is the codegen and runtime: pyspark/{constraint_dispatch,check_builder,check_ir,schema_builder,renderer,test_renderer,pipeline}.py, the path IR in overture-schema-system's field_path.py, and the runtime in overture-schema-pyspark/src/overture/schema/pyspark/.
• History is six commits: three supporting preludes (testmon for incremental test runs, the VehicleSelectorBase extraction, an example-data refresh), a requires-python packaging fix, the main pyspark feature commit, and the Java 17 CI pin. Build/CI hardening (unconditional test-all, the check job triggering on the Makefile/workflow themselves, typing-extensions declared) rides along in the feature commit.

Closes #517.

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/569/schema/index.html
🕐 Updated	Jul 17, 2026 19:53 UTC
📝 Commit	31d4b97
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.