fix: dtype-blind date coercion, silent field drop on tagging, id()-based measure names - #293
Draft
hussainsultan wants to merge 1 commit into
Draft
fix: dtype-blind date coercion, silent field drop on tagging, id()-based measure names#293hussainsultan wants to merge 1 commit into
hussainsultan wants to merge 1 commit into
Conversation
…sed names
Three defects found by probing invariants against the current branch.
JSON filters coerced any complete-ISO string to a date literal without
looking at the column, so a string column holding ISO-looking text (a
"2024-01-01" batch label) could not be filtered at all — the comparison
was rebuilt as string-vs-timestamp and the backend rejected it. Coercion
now requires a temporal column, and like/ilike never coerce since their
right operand is a string pattern. The coercion that Athena-style
backends need is unchanged, and an unknown dtype still coerces: only a
positively non-temporal column suppresses it.
Tagging dropped a whole field set when one field failed to serialize.
`value_or({})` turned "this measure holds a Python set" into
`measures: ()`, so the model tagged fine, reconstructed with no measures
at all, and only failed later as "Column 'total' is not found" — blaming
the query rather than the field. Same for dimensions, and calc measures
were skipped one at a time in both directions. All of these now name the
field that could not be written. This mattered more after the previous
commit's callable allowlist, which makes more expressions unserializable.
Positional measures were named `_measure_{id(item)}`, so the result
column was a memory address: it changed between runs and it reached the
xorq tag, making the metadata for one query differ per process. Names
are now positional (`_measure_0`), skipping any name a keyword measure
already claims — a collision the id() scheme was accidentally immune to.
Note on the cache key: a plain xorq memtable already hashes differently
across processes with no BSL involvement, so this fix removes BSL's
contribution to tag instability but does not by itself make memtable
expressions cacheable.
1292 tests pass; 15 added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #292 — review that one first; this PR's base is
fix/serialization-trust-boundary, so the diff here is only the three new fixes.Found by probing invariants against the branch state rather than by reading: 23 invariant checks over filtering, aggregation,
join_oneneutrality, and serialization round-trip fidelity. 20 held; the 3 that failed are fixed here. 1292 tests pass, 15 added.1. Date-literal coercion ignored the column's type (medium)
_convert_literalcoerced any complete-ISO string to a date/timestamp literal with no idea what it was being compared against. A string column holding ISO-looking text — abatchlabel of"2024-01-01"— could not be filtered at all:likewas worse: the pattern was coerced to a timestamp and then handed to.like(), failing signature validation. Coercion now requires a temporal column, and pattern operators never coerce. The behaviour the coercion exists for (backends like Athena needing typed date literals) is unchanged, and an unknown dtype still coerces — only a positively non-temporal column suppresses it.Worth noting
inwas unaffected, which is why this looked narrower than it was.2. One unserializable field emptied its whole field set (high)
_extract_semantic_tableusedvalue_or({})for dimensions, measures and calc measures. So a single field that could not be serialized — a measure holding a Pythonset, say — producedmeasures: ()in the tag. Tagging succeeded, the model reconstructed with no measures at all, and the failure surfaced much later as:which points at the query rather than at the field that could not be written. Calc measures had the same problem one entry at a time, in both directions. Every one of these now names the offending field at write time.
This interacts with #292: the callable allowlist there makes more expressions legitimately unserializable, so the blast radius of this bug had just grown.
3. Positional measures were named from
id()(medium)aggregate(lambda t: t.amount.sum())produced a column called_measure_4887309152— a memory address. It changed between runs, and because the name reaches the xorq tag, the metadata for one identical query differed per process. Names are now positional (_measure_0), skipping any name a keyword measure already claims.That last part matters:
aggs.update(aliased)runs after the positional loop, soaggregate(lambda t: ..., _measure_0=lambda t: ...)silently dropped the positional measure once names became predictable — a collision theid()scheme was accidentally immune to. Covered by a test.Correction worth recording: I initially attributed cross-process cache misses to this name. That was wrong — a plain xorq memtable hashes differently across processes with no BSL involvement at all. This fix removes BSL's contribution to tag instability (metadata is now byte-identical for the same query, where before it embedded an address) but does not by itself make memtable-backed expressions cacheable.
Invariants that held
Recording these because they are evidence, not decoration: sum over group-by parts equals the ungrouped total;
order_by/limitdo not change measure values;mean == sum/count;nuniqueon a unique column equalscount; an empty filter gives count 0;join_oneon a unique key leaves left-side sum/mean/nunique untouched; filters, callable order-by keys, limit/offset and calc measures all survive a tag round-trip;nest=inner values sum to their outer totals; and a dimension namedx.totalis not misrouted to HAVING by suffix matching.🤖 Generated with Claude Code