Skip to content

TCP J5: Add JSON support (String serialization) - #562

Open
alex-clickhouse wants to merge 2 commits into
tcp/epic-n11-query-parametersfrom
tcp/epic-j5-json
Open

TCP J5: Add JSON support (String serialization)#562
alex-clickhouse wants to merge 2 commits into
tcp/epic-n11-query-parametersfrom
tcp/epic-j5-json

Conversation

@alex-clickhouse

Copy link
Copy Markdown
Collaborator

Adds the ClickHouse JSON column type over the native protocol, stacked on #561 (query parameters).

JSON has several wire encodings. This reads and writes only the String serialization (wire version 1): a UInt64 version word, then a plain String column whose values are compact JSON text. So JsonStringColumnCodec is that version marker in front of StringColumnCodec, and a JSON column surfaces as IColumn<string>. Any other version (0 = V1, 2 = V2, 3 = FLATTENED, 4 = V3 — the per-path encodings) is rejected as a protocol error that names the setting to enable.

The per-path encoding (Tier 2) is deliberately not here. It splits the column into one sub-column per JSON path, and the path set lives in the per-block prefix and follows the data: one JSON(a UInt32, b String) column was seen to report dynamic path x in block 1 and y in block 2. A JSON column therefore has no stable schema across a result set, which is a real question for the borrowed-column API rather than a wire question. Tracked as J6 in the TODO, with the byte layout already written up.

What's here

  • Read + write of top-level JSON. The unit-test byte fixtures are real 26.6 server captures.
  • Every type-string spelling resolves to the same codec — JSON, JSON(a UInt32, b String), JSON(max_dynamic_paths=8), JSON(`b.c` String), JSON(SKIP z), JSON(a UInt32, SKIP REGEXP '^tmp(x,y)'). The arguments are deliberately not validated: the String tier ignores all of them, so a rejection would only break on a server that adds a new hint form. TypeParser already handled every form, so nothing changed there.
  • The client injects output_format_native_write_json_as_string = 1 next to the existing flattened flag, so a caller never has to know about it; a caller-supplied value still wins. Turning it off is how you would ask for an encoding this client cannot decode, so the override has to reach the server.
  • Composition, both directions: Nullable(JSON), Array(JSON), Tuple(JSON, String), Map(String, JSON), Nested(a JSON, b String), Variant(JSON, UInt64). Registering the codec also makes JSON reachable as a Dynamic runtime type, where the version word nests inside the Dynamic prefix after the type-name list.

Three points the wire behavior forced

  • The two directions are asymmetric. Reading needs the setting above, because the server otherwise sends a per-path encoding. Writing needs no setting at all — there is no input_format_native_* counterpart, because the server reads whichever version the prefix declares and parses the text into the real paths, typed paths included. Verified by hand-building a Native block and POSTing it.
  • NullPlaceholder is "{}", not the empty string. Nullable(JSON) is a legal type, and the server parses a JSON value instead of copying it, so it parses the placeholder at a NULL position too. An empty string is refused as INCORRECT_DATA. This is the first type whose placeholder is not inert, so IColumnCodec no longer promises that the server discards those bytes. The server writes {} there itself.
  • The codec must not implement ISpanWritableCodec<string>, although it delegates to StringColumnCodec, which does. That interface means "no state prefix", and a concatenating composite uses it to drive the inner codec one row at a time. The version word would be dropped — and only on the ergonomic path, since the dense branch ignores the interface — which desynchronizes Array(JSON). A test fails if anyone adds the interface.

Testing

  • Codec unit tests (24): documented bytes from a server capture, version-not-1 rejection for all four per-path versions, the eight type-string forms, CanWrite, the {} placeholder, Nullable(JSON) bytes (version ahead of the null map), Array(JSON) bytes (version once, ahead of the offsets), zero rows, and per AGENTS.local.md three start > 0 slice tests — the ergonomic column, the dense read-back, and an Array(JSON) whose offsets must rebase to the slice start rather than carry the column's running total.
  • Integration round-trip cases (11), which fan out through five fixtures — round-trip, dense re-insert, one-row-per-block slicing, POCO read, POCO write. A JSON column maps to a string POCO property with no registration work.
  • One case pins the server's own normalization, so text in is not assumed to be text out: keys sorted ordinally, whitespace dropped, numbers re-rendered (1e31000), a dotted key read as nesting, a JSON null or an empty object contributing no path, and a declared typed path defaulting when the value omits it.
  • Full suite green: 2598 tests. JsonStringColumnCodec fully covered.

Notes / deferred

  • Variant(JSON, String) is ambiguous on the ergonomic write path. Both alternatives surface CLR string, and that path picks an alternative by runtime CLR type, so every string takes the first-listed arm (JSON sorts first) and a non-JSON string is refused server-side. Reading works, and the dense VariantColumn source is unaffected because it carries explicit discriminators. Same root cause as the outer-type-name matching gap in the N11 table; logged in the TODO as needing a semantic decision. The corpus covers Variant(JSON, UInt64), whose alternatives are CLR-distinguishable.
  • Dynamic holding JSON is read-only. DynamicTypeInference maps a CLR string to String, not JSON, so that shape cannot be written ergonomically. Covered by a read test.
  • LowCardinality(JSON) needs nothing: the server rejects the type.
  • No CHANGELOG entry — the TCP client is [Experimental] and this branch keeps TCP work out of it.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds native TCP support for ClickHouse JSON columns using String serialization (wire version 1).

Changes:

  • Adds JSON codec registration, serialization, and protocol validation.
  • Automatically requests JSON-as-String output.
  • Adds comprehensive codec and integration coverage.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
IColumnCodec.cs Clarifies nullable placeholders.
ColumnCodecRegistry.cs Registers JSON codec.
JsonStringColumnCodec.cs Implements JSON String serialization.
ClickHouseTcpClient.cs Injects JSON serialization setting.
InsertRoundTripCase.cs Adds JSON round-trip cases.
JsonStringColumnCodecTests.cs Tests wire behavior and composition.
ClickHouseTcpClientIntegrationTests.cs Tests automatic JSON decoding.
ClickHouseTcpClientSettingsTests.cs Tests setting injection and overrides.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ClickHouse.Driver.Tcp/Client/ClickHouseTcpClient.cs
@alex-clickhouse
alex-clickhouse marked this pull request as ready for review August 16, 2026 17:46
Reads and writes the ClickHouse `JSON` column in its String serialization
(wire version 1): a `UInt64` version word, then a plain String column whose
values are compact JSON text. `JsonStringColumnCodec` is that version marker
in front of `StringColumnCodec`, and surfaces `IColumn<string>`. Any other
version (0 = V1, 2 = V2, 3 = FLATTENED, 4 = V3 — the per-path encodings) is
rejected as a protocol error that names the setting.

The type string does not change this layout, so every spelling resolves to
the same codec. Its arguments are deliberately not validated: the String tier
ignores all of them, and a rejection would only break on a server that adds
a new hint form.

Three points that the wire behavior forced:

* The two directions are asymmetric. To read, the server must send text, so
  the client injects `output_format_native_write_json_as_string = 1` next to
  the flattened flag. A caller value still wins. To write, no setting is
  needed: the server reads the version that the prefix declares, and parses
  the text into the real paths, typed paths included.

* `NullPlaceholder` is `"{}"`, not the empty string. `Nullable(JSON)` is a
  legal type, and the server parses a JSON value instead of copying it, so
  it also parses the placeholder at a NULL position. An empty string is
  refused as INCORRECT_DATA. This is the first type whose placeholder is not
  inert, so `IColumnCodec` no longer promises that the server discards
  those bytes.

* The codec must not implement `ISpanWritableCodec<string>`, although it
  delegates to `StringColumnCodec`, which does. That interface means "no
  state prefix", and a concatenating composite uses it to drive the inner
  codec one row at a time. The version word would be lost, and only on the
  ergonomic path, which desynchronizes `Array(JSON)`.

Co-Authored-By: Claude <noreply@anthropic.com>
The corpus gained Array(Nested(a UInt8)), Tuple(Nested(a UInt8), String),
Map(String, Nested(a UInt8)) and Map(Nested(a UInt8), UInt32). The POCO write
test runs the whole corpus, so those four enrolled themselves in it.

None of them can be gathered from a property, for the same reason a top-level
Nested cannot: the codec writes from flat field columns behind shared offsets,
which no property holds. The branch that says so matched StartsWith("Nested("),
so it caught only the top-level shape and let the four new cases reach the
happy path, where they failed on the very refusal the branch expects.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants