TCP J5: Add JSON support (String serialization) - #562
Open
alex-clickhouse wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds native TCP support for ClickHouse JSON columns using String serialization (wire version 1).
Changes:
- Adds JSON codec registration, serialization, and protocol validation.
- Automatically requests JSON-as-String output.
- Adds comprehensive codec and integration coverage.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
IColumnCodec.cs |
Clarifies nullable placeholders. |
ColumnCodecRegistry.cs |
Registers JSON codec. |
JsonStringColumnCodec.cs |
Implements JSON String serialization. |
ClickHouseTcpClient.cs |
Injects JSON serialization setting. |
InsertRoundTripCase.cs |
Adds JSON round-trip cases. |
JsonStringColumnCodecTests.cs |
Tests wire behavior and composition. |
ClickHouseTcpClientIntegrationTests.cs |
Tests automatic JSON decoding. |
ClickHouseTcpClientSettingsTests.cs |
Tests setting injection and overrides. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
alex-clickhouse
marked this pull request as ready for review
August 16, 2026 17:46
Reads and writes the ClickHouse `JSON` column in its String serialization
(wire version 1): a `UInt64` version word, then a plain String column whose
values are compact JSON text. `JsonStringColumnCodec` is that version marker
in front of `StringColumnCodec`, and surfaces `IColumn<string>`. Any other
version (0 = V1, 2 = V2, 3 = FLATTENED, 4 = V3 — the per-path encodings) is
rejected as a protocol error that names the setting.
The type string does not change this layout, so every spelling resolves to
the same codec. Its arguments are deliberately not validated: the String tier
ignores all of them, and a rejection would only break on a server that adds
a new hint form.
Three points that the wire behavior forced:
* The two directions are asymmetric. To read, the server must send text, so
the client injects `output_format_native_write_json_as_string = 1` next to
the flattened flag. A caller value still wins. To write, no setting is
needed: the server reads the version that the prefix declares, and parses
the text into the real paths, typed paths included.
* `NullPlaceholder` is `"{}"`, not the empty string. `Nullable(JSON)` is a
legal type, and the server parses a JSON value instead of copying it, so
it also parses the placeholder at a NULL position. An empty string is
refused as INCORRECT_DATA. This is the first type whose placeholder is not
inert, so `IColumnCodec` no longer promises that the server discards
those bytes.
* The codec must not implement `ISpanWritableCodec<string>`, although it
delegates to `StringColumnCodec`, which does. That interface means "no
state prefix", and a concatenating composite uses it to drive the inner
codec one row at a time. The version word would be lost, and only on the
ergonomic path, which desynchronizes `Array(JSON)`.
Co-Authored-By: Claude <noreply@anthropic.com>
alex-clickhouse
force-pushed
the
tcp/epic-j5-json
branch
from
August 16, 2026 17:52
c6c242a to
e8cd96a
Compare
The corpus gained Array(Nested(a UInt8)), Tuple(Nested(a UInt8), String),
Map(String, Nested(a UInt8)) and Map(Nested(a UInt8), UInt32). The POCO write
test runs the whole corpus, so those four enrolled themselves in it.
None of them can be gathered from a property, for the same reason a top-level
Nested cannot: the codec writes from flat field columns behind shared offsets,
which no property holds. The branch that says so matched StartsWith("Nested("),
so it caught only the top-level shape and let the four new cases reach the
happy path, where they failed on the very refusal the branch expects.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the ClickHouse
JSONcolumn type over the native protocol, stacked on #561 (query parameters).JSONhas several wire encodings. This reads and writes only the String serialization (wire version 1): aUInt64version word, then a plain String column whose values are compact JSON text. SoJsonStringColumnCodecis that version marker in front ofStringColumnCodec, and a JSON column surfaces asIColumn<string>. Any other version (0= V1,2= V2,3= FLATTENED,4= V3 — the per-path encodings) is rejected as a protocol error that names the setting to enable.The per-path encoding (Tier 2) is deliberately not here. It splits the column into one sub-column per JSON path, and the path set lives in the per-block prefix and follows the data: one
JSON(a UInt32, b String)column was seen to report dynamic pathxin block 1 andyin block 2. A JSON column therefore has no stable schema across a result set, which is a real question for the borrowed-column API rather than a wire question. Tracked as J6 in the TODO, with the byte layout already written up.What's here
JSON. The unit-test byte fixtures are real 26.6 server captures.JSON,JSON(a UInt32, b String),JSON(max_dynamic_paths=8),JSON(`b.c` String),JSON(SKIP z),JSON(a UInt32, SKIP REGEXP '^tmp(x,y)'). The arguments are deliberately not validated: the String tier ignores all of them, so a rejection would only break on a server that adds a new hint form.TypeParseralready handled every form, so nothing changed there.output_format_native_write_json_as_string = 1next to the existing flattened flag, so a caller never has to know about it; a caller-supplied value still wins. Turning it off is how you would ask for an encoding this client cannot decode, so the override has to reach the server.Nullable(JSON),Array(JSON),Tuple(JSON, String),Map(String, JSON),Nested(a JSON, b String),Variant(JSON, UInt64). Registering the codec also makes JSON reachable as aDynamicruntime type, where the version word nests inside theDynamicprefix after the type-name list.Three points the wire behavior forced
input_format_native_*counterpart, because the server reads whichever version the prefix declares and parses the text into the real paths, typed paths included. Verified by hand-building a Native block and POSTing it.NullPlaceholderis"{}", not the empty string.Nullable(JSON)is a legal type, and the server parses a JSON value instead of copying it, so it parses the placeholder at a NULL position too. An empty string is refused asINCORRECT_DATA. This is the first type whose placeholder is not inert, soIColumnCodecno longer promises that the server discards those bytes. The server writes{}there itself.ISpanWritableCodec<string>, although it delegates toStringColumnCodec, which does. That interface means "no state prefix", and a concatenating composite uses it to drive the inner codec one row at a time. The version word would be dropped — and only on the ergonomic path, since the dense branch ignores the interface — which desynchronizesArray(JSON). A test fails if anyone adds the interface.Testing
CanWrite, the{}placeholder,Nullable(JSON)bytes (version ahead of the null map),Array(JSON)bytes (version once, ahead of the offsets), zero rows, and perAGENTS.local.mdthreestart > 0slice tests — the ergonomic column, the dense read-back, and anArray(JSON)whose offsets must rebase to the slice start rather than carry the column's running total.stringPOCO property with no registration work.1e3→1000), a dotted key read as nesting, a JSONnullor an empty object contributing no path, and a declared typed path defaulting when the value omits it.JsonStringColumnCodecfully covered.Notes / deferred
Variant(JSON, String)is ambiguous on the ergonomic write path. Both alternatives surface CLRstring, and that path picks an alternative by runtime CLR type, so every string takes the first-listed arm (JSONsorts first) and a non-JSON string is refused server-side. Reading works, and the denseVariantColumnsource is unaffected because it carries explicit discriminators. Same root cause as the outer-type-name matching gap in the N11 table; logged in the TODO as needing a semantic decision. The corpus coversVariant(JSON, UInt64), whose alternatives are CLR-distinguishable.Dynamicholding JSON is read-only.DynamicTypeInferencemaps a CLR string toString, notJSON, so that shape cannot be written ergonomically. Covered by a read test.LowCardinality(JSON)needs nothing: the server rejects the type.[Experimental]and this branch keeps TCP work out of it.🤖 Generated with Claude Code