Fix Cavena890 Chinese write path so text and italics round-trip#12594
Merged
Conversation
The Chinese text field is UTF-16BE - the reader trims 0x00/0x7F fillers, forces an even length, and decodes the field as byte pairs. The writer however emitted only the high byte of each character (encoding.GetBytes(...)[0]) and wrote lone italic-marker bytes without advancing the index, so no Chinese text or formatting could survive a save/load cycle. Writer: emit the full UTF-16BE pair for every character, and write the italic markers as in-band 0x00 0x88 / 0x00 0x98 pairs so the byte pairing stays aligned; all writes are bounds-checked against the 51-byte field. Reader: match the decoded marker/filler chars explicitly (U+0088, U+0098, U+007F, U+00BE) instead of via Encoding.Default.GetString(..), which is platform dependent - on .NET Core (UTF-8) 0x88/0x98/0xBE all decode to U+FFFD, so any undecodable byte in a file was turned into "<i>". Round-trip tests added (plain text, two lines, italics); both fail against the old code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the #12580 review, where the Chinese
index++commit was dropped because the whole Chinese write path was found to be unable to round-trip. This fixes the path properly.The problem: the Chinese text field is UTF-16BE —
FixTexttrims the 0x00/0x7F fillers, forces an even length, and decodes the field withEncoding.GetEncoding(1201)as byte pairs. The writer, however, emitted only the high byte of each character (encoding.GetBytes(new[] { current })[0]) and wrote lone italic-marker bytes without advancing the index (so the next character overwrote them). No Chinese text, let alone italics, could survive a save/load cycle.Writer (
GetTextAsBytes): every character now writes its full UTF-16BE pair, and the italic markers are written as in-band0x00 0x88/0x00 0x98pairs so the byte pairing stays aligned. All writes are bounds-checked against the 51-byte field (silent truncation, as before).Reader (
FixText, Chinese branch): the marker/filler comparisons usedEncoding.Default.GetString(..), which is platform dependent — on .NET CoreEncoding.Defaultis UTF-8, where0x88/0x98/0xBEeach decode to U+FFFD, meaning any undecodable byte in a real file was replaced with<i>. The comparisons now match the actual decoded chars (U+0088, U+0098, U+007F, U+00BE) explicitly and deterministically.Compatibility notes:
0x00 0x88marker-pair layout is the natural in-band encoding under that pairing, but I could not verify it against files produced by real Cavena tools (no sample.890files in the repo). Since the old marker matching couldn't fire deterministically on any platform, no working real-file italic behavior is lost either way.Tests: save→load round-trip through the public
Save/LoadSubtitleAPI for plain Chinese text, a two-line paragraph, and an italicized line. Both tests fail against the old code. All 531 libse and 237 seconv tests pass.🤖 Generated with Claude Code