fix: escape/unescape UTF-8 in FAST string codec#4
Merged
Conversation
Legacy fast string encoding ORs 0x80 onto the final byte and clears bit 7 on decode, so any UTF-8 payload (e.g. "café") truncates, mutates, and desynchronizes the FSON stream — bricking YarDB files after create. Keep the ASCII terminator codec for existing databases; emit length- prefixed utf8 / name_utf8 tags for high-bit content.
Replace encode-time utf8/name_utf8 tags with Escape/unEscape so high-bit content rides the existing string/name types through the 7-bit FAST terminator codec. Ordinary ASCII stays bit-identical on the wire; the decoder still accepts the intermediate length-prefixed utf8 tags. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
Drop utf8/name_utf8 tags and length-prefixed encode_bytes. The fast string codec stays the original 7-bit terminator write/read; FSON Escape/unEscape alone make UTF-8 safe for that path. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
Follow project naming: snake_case for all functions. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
cursor
Bot
changed the title
Move escape/unescape into xson:fast so ASCII strings keep the original no-temporary path. High-bit content is streamed on write and unescaped in place on read; FSON string/name handling is unchanged at the call site. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
cursor
Bot
changed the title
One loop emits ASCII as-is and only takes the ESC+hex branch for high-bit / escape-marker bytes. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
Use std::uint8_t (byte), named literals, and put_byte/to_char/to_byte so high-bit math stays unsigned without static_cast noise at every use site. Co-authored-by: Kaius Ruokonen <ruoka@users.noreply.github.com>
ruoka
marked this pull request as ready for review
July 17, 2026 20:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FSON string/name payloads used the FAST 7-bit terminator codec (
last_byte | 0x80/ clear bit 7 on decode), which corrupts UTF-8 and can desynchronize the stream.Approach: fold high-bit handling into
xson:faststring encode/decode (no temporary escaped string, noneeds_escapepre-scan):ESC + two hex digitsbranch; terminator bit on the final emitted bytebyte(std::uint8_t) with named literals (escape_marker,stop_bit, …) andput_byte/to_char/to_byteat the char boundaryfast::encode/fast::decodeValidation
./tools/CB.sh debug test "String|UTF-8|escape"→ 49/49 passed