TCP N5a: Add the read-side mirror of WritableElementTypes to IColumnCodec - #548
Draft
alex-clickhouse wants to merge 2 commits into
Draft
TCP N5a: Add the read-side mirror of WritableElementTypes to IColumnCodec#548alex-clickhouse wants to merge 2 commits into
alex-clickhouse wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds read-side type projections to TCP codecs, preparing POCO reads on top of #462.
Changes:
- Adds
ReadableElementTypesand expression-basedProjectRead. - Adds shared date/time projections and nullable/low-cardinality lifting.
- Adds unit and server integration coverage.
The documented reference-source-to-nullable-value projection case remains broken and requires correction.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
Types/IColumnCodec.cs |
Defines the projection contract. |
Types/ColumnValueProjections.cs |
Centralizes projection logic. |
Types/DateTimeColumn.cs |
Reuses shared conversion. |
Types/DateTime64Column.cs |
Reuses shared scaled conversion. |
Types/TimeColumn.cs |
Reuses shared duration conversion. |
Types/Time64Column.cs |
Reuses shared scaled conversion. |
Codecs/DateTimeColumnCodec.cs |
Advertises and builds date projections. |
Codecs/DateTime64ColumnCodec.cs |
Adds scaled date projections. |
Codecs/TimeColumnCodec.cs |
Adds TimeSpan projection. |
Codecs/Time64ColumnCodec.cs |
Adds scaled TimeSpan projection. |
Codecs/NullableColumnCodec.cs |
Lifts inner readable types. |
Codecs/LowCardinalityColumnCodec.cs |
Propagates inner projections. |
ColumnReadProjectionTests.cs |
Tests contracts and expressions. |
ColumnReadProjectionIntegrationTests.cs |
Validates projections against ClickHouse. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+178
to
+180
| return innerTarget.IsValueType | ||
| ? ColumnValueProjections.ProjectNullable(value, inner, innerTarget, targetType) | ||
| : inner.ProjectRead(value, innerTarget); |
Comment on lines
+382
to
+384
| return nullable && innerTarget.IsValueType | ||
| ? ColumnValueProjections.ProjectNullable(value, inner, innerTarget, targetType) | ||
| : inner.ProjectRead(value, innerTarget); |
alex-clickhouse
force-pushed
the
tcp/epic-n5-poco
branch
from
August 14, 2026 16:44
bb4900b to
2602deb
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
alex-clickhouse
force-pushed
the
tcp/epic-n5-poco
branch
from
August 15, 2026 11:48
2602deb to
4435f57
Compare
alex-clickhouse
force-pushed
the
tcp/epic-n5-poco
branch
from
August 16, 2026 10:31
4435f57 to
9e22d63
Compare
alex-clickhouse
force-pushed
the
tcp/epic-n5-poco
branch
from
August 16, 2026 17:50
9e22d63 to
735a6a8
Compare
A codec already declares which CLR types it accepts on the write path (WritableElementTypes, CanWrite, NullPlaceholderAs), and converts them inside WriteColumn. The read path had no counterpart: a codec decodes to exactly one canonical type, and for the date/time family that type is the raw wire count (DateTime is IColumn<uint>, DateTime64 is IColumn<long>). So the most common POCO member type has no match at all. Add the mirror rather than a third home for the same knowledge: - ReadableElementTypes, defaulting to [ElementType], mirroring WritableElementTypes. - ProjectRead(Expression value, Type targetType), mirroring WriteColumn's type switch. An expression, not a delegate, so a caller compiling a per-column read loop can inline the conversion instead of paying an indirect call per row. Per-column state (scale, timezone) is embedded as a constant, so the result closes over nothing. Overridden by the four codecs whose canonical type is a raw count: DateTime, DateTime64, Time, Time64. Nullable and LowCardinality delegate to their inner codec and lift through Nullable<> where the surface wrapped the inner type. Enum is deliberately excluded: it does not override WritableElementTypes either, so offering a label on read would be a read-only asymmetry rather than a mirror. The conversions move into ColumnValueProjections, which the four columns now share instead of each keeping a private copy, so a raw count has one calendar reading whichever surface asks. TimeColumn's projection changes spelling (TimeSpan.FromSeconds to FromTicks) and is value-identical over the type's whole range. The DateTime projection follows the HTTP driver's Kind rule: a zero offset yields Kind=Utc, any other offset the wall clock as Unspecified. The two clients still diverge for a bare DateTime, where HTTP presents in UTC and this client resolves the session timezone; that is documented, not changed. Nothing consumes this yet, so the tests carry it: the codecs' behavior is pinned against independently derived constants, against a real server's own timezone and scale handling, and by an invariant sweep asserting every registered codec leads its readable list with ElementType and can project each type it advertises.
The divergence from the HTTP driver on a zone-less DateTime is deliberate: this client resolves session_timezone, so it agrees with what the server would display, at the cost of a POCO moved from the HTTP client shifting by the session offset. Say so, so it is not later 'fixed' toward HTTP. Comment only; the behavior is unchanged.
alex-clickhouse
force-pushed
the
tcp/epic-n5-poco
branch
from
August 17, 2026 09:49
735a6a8 to
48f9a17
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #462 (
tcp/epic-n1-client) — review that first; this PR's diff is the last commit only.First step of the Branch 2 POCO epic (N5a/N5/N9), landed on its own because it changes an
internalinterface every codec implements and nothing consumes it yet.Why
A codec already declares what it accepts on the write path —
WritableElementTypes(preference-ordered),CanWriteas the membership test,NullPlaceholderAs, and the type switch insideWriteColumn.DateTimeColumnCodeceven already holds theTimeZoneInfoand theToUtc/ToUnixSecondshelpers.The read path had no counterpart, because a codec decodes to exactly one canonical type — and for the date/time family that type is the raw wire count:
IColumn<T>DateTimeuint(epoch seconds)DateTime64(s)long(raw count at scale)Time/Time64(s)int/longSo
public DateTime CreatedAt { get; set; }— the most common POCO member there is — has no match at all. Rather than invent a third home for knowledge the write path already keeps on the codec, this adds the mirror.What
ReadableElementTypes— defaults to[ElementType], mirroringWritableElementTypes.ProjectRead(Expression value, Type targetType)— mirrorsWriteColumn's type switch. AnExpression, not aFunc<,>, so the compiled per-column read loop the POCO reader will emit can inline the conversion; a delegate would cost an indirect call per row. Scale and timezone are embedded as constants, so the result closes over nothing.Overridden by the four raw-count codecs.
NullableandLowCardinalitydelegate to their inner codec and lift throughNullable<>where the surface wrapped the inner type.The conversions move into a new
ColumnValueProjectionsthatDateTimeColumn,DateTime64Column,TimeColumnandTime64Columnnow share instead of each keeping a private copy, so a raw count has one calendar reading whichever surface asks for it.Deliberately excluded
Enum.
EnumColumnCodec<T>does not overrideWritableElementTypeseither — write takes only the raw ordinal — so offering astringlabel on read would be a read-only asymmetry rather than a mirror. Pinned by a test so adding it later is a visible choice. Label mapping stays possible (the label maps are reachable only from the codec) but is now its own decision.Composites.
Array/Map/Tuple/Nesteddo not lift their children's readable types, soArray(DateTime)reads only asuint[]. This is faithful: their write contracts do not lift either (ArrayColumnCodec.CanWriteiscolumn is IColumn<TElement[]>over the child's canonicalElementType), so the gap is symmetric today. Lifting it is not a small extension of this contract — the readable set of a composite is the cartesian product of its children's, soTupleof sevenDateTime64s would enumerate 3^7 entries. See the discussion below.DateTimeKind semanticsPresentAsDateTimefollows the HTTP driver'sAbstractDateTimeType.ToDateTime: a zero offset yieldsKind=Utc, any other offset the wall clock in the column's zone asKind=Unspecified.For a bare
DateTime/DateTime64the two clients deliberately differ: HTTP presents it in UTC because its type object carries no zone, whereas this client resolvessession_timezoneand so agrees with what the server itself would display. That was weighed and chosen — keeping the whole TCP read path honoringsession_timezonebeat cross-client parity, at the cost of a POCO moved from the HTTP client shifting by the session offset on bare columns. Columns whose type names a timezone match HTTP exactly. Recorded in thePresentAsDateTimedoc comment so it is not later "fixed" toward HTTP.Adjacent and pre-existing: reads use the session timezone while writes use
ResolveContext.ForWrite, which carries none, soToUtcreads anUnspecifiedDateTimeas a UTC wall clock. A round trip through a bare column on a non-UTC session is therefore not the identity.Tests
Nothing consumes this yet, so the tests carry the PR.
ColumnReadProjectionTests) — expected values derived independently, not from the implementation:1700000000=2023-11-14T22:13:20Z, Berlin+01:00in November and+02:00for a July instant (a fixed-base-offset implementation passes the first and fails the second), scale-9…123456789truncating to.1234567,Time64(9)-1000000001→-00:00:01showing truncation toward zero.ElementType, lists no duplicate, and can actually project each type it advertises — so a codec cannot advertise a projection it does not have.HasValue,Value), so it binds a local first. Verified by mutation: inlining the source twice fails this test.ColumnReadProjectionIntegrationTests) — projections checked against a real server's own timezone and scale handling, including a DST instant andLowCardinality(Nullable(DateTime)).Coverage: every changed file ≥89.8%,
ColumnValueProjectionsandIColumnCodecat 100%. Full TCP suite 1394/1394 green.Review note
A review pass caught a latent bug, fixed here: both wrappers computed the advertised type per read type but decided whether to lift by testing the inner codec's canonical type. Those agree only while every read type shares its value/reference-ness with
ElementType— so the moment a reference-typed inner gained a value-typed reading (FixedString(16)→Guidbeing the plausible one),Nullable(FixedString(16))would advertiseGuid?and return aGuid-typed expression, silently turning a null row intodefault(Guid). The discriminator is now the type being projected to.Being straight about its test status: mutating that condition back leaves all tests passing, because no current type pair distinguishes the two. The guard test documents the invariant for when such a type is added rather than catching a live bug.
🤖 Generated with Claude Code