Skip to content

fix: dtype-blind date coercion, silent field drop on tagging, id()-based measure names - #293

Draft
hussainsultan wants to merge 1 commit into
fix/serialization-trust-boundaryfrom
fix/round7-defects
Draft

fix: dtype-blind date coercion, silent field drop on tagging, id()-based measure names#293
hussainsultan wants to merge 1 commit into
fix/serialization-trust-boundaryfrom
fix/round7-defects

Conversation

@hussainsultan

Copy link
Copy Markdown
Collaborator

Stacked on #292 — review that one first; this PR's base is fix/serialization-trust-boundary, so the diff here is only the three new fixes.

Found by probing invariants against the branch state rather than by reading: 23 invariant checks over filtering, aggregation, join_one neutrality, and serialization round-trip fidelity. 20 held; the 3 that failed are fixed here. 1292 tests pass, 15 added.

1. Date-literal coercion ignored the column's type (medium)

_convert_literal coerced any complete-ISO string to a date/timestamp literal with no idea what it was being compared against. A string column holding ISO-looking text — a batch label of "2024-01-01" — could not be filtered at all:

{"field": "batch", "operator": "=", "value": "2024-01-01"}
XorqTypeError: Arguments batch:string and Literal(2024-01-01 00:00:00):timestamp are not comparable

like was worse: the pattern was coerced to a timestamp and then handed to .like(), failing signature validation. Coercion now requires a temporal column, and pattern operators never coerce. The behaviour the coercion exists for (backends like Athena needing typed date literals) is unchanged, and an unknown dtype still coerces — only a positively non-temporal column suppresses it.

Worth noting in was unaffected, which is why this looked narrower than it was.

2. One unserializable field emptied its whole field set (high)

_extract_semantic_table used value_or({}) for dimensions, measures and calc measures. So a single field that could not be serialized — a measure holding a Python set, say — produced measures: () in the tag. Tagging succeeded, the model reconstructed with no measures at all, and the failure surfaced much later as:

XorqTypeError: Column 'total' is not found in table. Existing columns: 'a', 'b'.

which points at the query rather than at the field that could not be written. Calc measures had the same problem one entry at a time, in both directions. Every one of these now names the offending field at write time.

This interacts with #292: the callable allowlist there makes more expressions legitimately unserializable, so the blast radius of this bug had just grown.

3. Positional measures were named from id() (medium)

aggregate(lambda t: t.amount.sum()) produced a column called _measure_4887309152 — a memory address. It changed between runs, and because the name reaches the xorq tag, the metadata for one identical query differed per process. Names are now positional (_measure_0), skipping any name a keyword measure already claims.

That last part matters: aggs.update(aliased) runs after the positional loop, so aggregate(lambda t: ..., _measure_0=lambda t: ...) silently dropped the positional measure once names became predictable — a collision the id() scheme was accidentally immune to. Covered by a test.

Correction worth recording: I initially attributed cross-process cache misses to this name. That was wrong — a plain xorq memtable hashes differently across processes with no BSL involvement at all. This fix removes BSL's contribution to tag instability (metadata is now byte-identical for the same query, where before it embedded an address) but does not by itself make memtable-backed expressions cacheable.

Invariants that held

Recording these because they are evidence, not decoration: sum over group-by parts equals the ungrouped total; order_by/limit do not change measure values; mean == sum/count; nunique on a unique column equals count; an empty filter gives count 0; join_one on a unique key leaves left-side sum/mean/nunique untouched; filters, callable order-by keys, limit/offset and calc measures all survive a tag round-trip; nest= inner values sum to their outer totals; and a dimension named x.total is not misrouted to HAVING by suffix matching.

🤖 Generated with Claude Code

…sed names

Three defects found by probing invariants against the current branch.

JSON filters coerced any complete-ISO string to a date literal without
looking at the column, so a string column holding ISO-looking text (a
"2024-01-01" batch label) could not be filtered at all — the comparison
was rebuilt as string-vs-timestamp and the backend rejected it. Coercion
now requires a temporal column, and like/ilike never coerce since their
right operand is a string pattern. The coercion that Athena-style
backends need is unchanged, and an unknown dtype still coerces: only a
positively non-temporal column suppresses it.

Tagging dropped a whole field set when one field failed to serialize.
`value_or({})` turned "this measure holds a Python set" into
`measures: ()`, so the model tagged fine, reconstructed with no measures
at all, and only failed later as "Column 'total' is not found" — blaming
the query rather than the field. Same for dimensions, and calc measures
were skipped one at a time in both directions. All of these now name the
field that could not be written. This mattered more after the previous
commit's callable allowlist, which makes more expressions unserializable.

Positional measures were named `_measure_{id(item)}`, so the result
column was a memory address: it changed between runs and it reached the
xorq tag, making the metadata for one query differ per process. Names
are now positional (`_measure_0`), skipping any name a keyword measure
already claims — a collision the id() scheme was accidentally immune to.

Note on the cache key: a plain xorq memtable already hashes differently
across processes with no BSL involvement, so this fix removes BSL's
contribution to tag instability but does not by itself make memtable
expressions cacheable.

1292 tests pass; 15 added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant