Skip to content

Fix six silent text-corruption bugs in libse#12580

Merged
niksedk merged 7 commits into
SubtitleEdit:mainfrom
ivandrofly:fix/text-corruption-fixes
Jul 18, 2026
Merged

Fix six silent text-corruption bugs in libse#12580
niksedk merged 7 commits into
SubtitleEdit:mainfrom
ivandrofly:fix/text-corruption-fixes

Conversation

@ivandrofly

@ivandrofly ivandrofly commented Jul 17, 2026

Copy link
Copy Markdown
Member

Fixes six verified bugs that silently corrupt subtitle text, one commit each, plus tests pinning each fix:

  • DvdStudioPro: the allBoldItalic branch emitted ^U^B^I|^I^B^U (copy-paste of the underline+bold+italic template), injecting stray underline codes the reader does not parse, so they survived round-trip as literal ^U text; now ^B^I|^I^B.
  • FixStartWithUppercaseLetterAfterColon (Turkish): the fix operated at the colon's index instead of the letter's, so "Ali: iyi" became "Aliİ iyi"; now correctly "Ali: İyi".
  • FixUppercaseIInsideWords: off-by-one guard (Length - 3 instead of - 2) skipped the consonant check when the match ended the stripped text, so "You HIt." became "You Hlt."; now "You Hit.".
  • FileUtil.ReadAllLinesShared: tested the UTF-16 BE BOM (FE FF) for Encoding.Unicode (LE) — ReadAllTextShared directly below has it right — leaving U+FEFF on the first line of UTF-16LE files.
  • Utilities.SplitEndTags: appended stripped closing tags instead of prepending; since the loop walks right-to-left, post came back reversed, so <font color="white"><i>Hello</i></font> round-tripped through RemoveSymbols/AddSymbols/ToggleSymbols as invalid …Hello</font></i>.
  • Cavena890 (Hebrew): </i> set skipCount to 2 instead of 3, leaking a literal > byte into the output (the Chinese branch and the <i> handling show the intended pattern).

Dropped from the original version: the Cavena890 Chinese index++ commit. Review showed the whole Chinese write path cannot round-trip regardless: it writes one byte per character (encoding.GetBytes(...)[0] — the high byte of the UTF-16BE pair) while the reader decodes the buffer as UTF-16BE byte pairs, so adding index++ for the italic markers does not make italics (or any Chinese text) survive, and the "fix" claim was not verifiable. That path needs a real rewrite in a separate PR.

Test coverage (all verified to fail against the pre-fix code): SplitEndTags suffix order (nested tags + trailing space), FixUppercaseIInsideWords end-of-text consonant check plus mid-text control, UTF-16LE BOM strip, DvdStudioPro bold+italic encode (exact separator, no ^U) and a load/save round-trip, and the Turkish colon regression test.

All 529 libse tests pass.

🤖 Generated with Claude Code

@niksedk
niksedk requested a review from Copilot July 18, 2026 07:26

Copilot AI commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

@niksedk Unfortunately, I hit an error while trying to use the custom Copilot setup steps configured for this repository and had to close this PR. The error I am seeing is:

no `copilot-setup-steps` job found in your `copilot-setup-steps.yml` workflow file. Please ensure you have a single job named `copilot-setup-steps`. For more details, see https://gh.io/copilot/actions-setup-steps

Once you or someone with the necessary access fixes the problem, please unassign and then reassign issue #0 to me and I'll retry. Thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses multiple verified cases where subtitle text could be silently corrupted during format conversion or during “Fix common errors” text transformations in libse, and adds a regression test to prevent reintroducing the Turkish colon-capitalization bug.

Changes:

  • Fixes incorrect style/control-code emission and buffer indexing in specific subtitle format encoders (DvdStudioPro, Cavena890) to prevent stray markers or dropped italics.
  • Corrects several off-by-one / wrong-index text transformation bugs (Turkish post-colon capitalization; uppercase “I” inside words).
  • Fixes round-trip corruption in shared utilities (end-tag splitting order; UTF-16LE BOM handling) and adds a focused Turkish regression test.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/libse/Forms/FixCommonErrors/FixStartWithUppercaseLetterAfterColonTest.cs Adds a Turkish regression test ensuring capitalization occurs at the letter position (not overwriting the colon).
src/libse/SubtitleFormats/DvdStudioPro.cs Removes incorrect underline control codes from the “all bold+italic” newline template to avoid stray ^U artifacts.
src/libse/SubtitleFormats/Cavena890.cs Fixes </i> skip length for Hebrew and ensures Chinese italic marker bytes advance the output index so they aren’t overwritten.
src/libse/Forms/FixCommonErrors/FixUppercaseIInsideWords.cs Fixes an off-by-one guard so the consonant check runs when the match ends near the end of the string.
src/libse/Forms/FixCommonErrors/FixStartWithUppercaseLetterAfterColon.cs Fixes Turkish casing by replacing the character at k (the letter) rather than j (the colon).
src/libse/Common/Utilities.cs Prepends stripped end-tags/spaces into post to preserve correct closing-tag order on round-trip.
src/libse/Common/FileUtil.cs Corrects UTF-16LE BOM detection for Encoding.Unicode to avoid leaving U+FEFF at the start of the first line.

@niksedk

niksedk commented Jul 18, 2026

Copy link
Copy Markdown
Member

AI-generated bug fix pull requests

For future AI-generated bug fix PRs, please limit each pull request to 1–3 related fixes (depending on their size and complexity). Smaller, focused PRs are much easier to review, test, and merge.

Please also include a brief manual comment describing how you personally verified each fix. AI-generated code can contain mistakes or unintended side effects, so every submitted change should be tested and validated by the contributor before opening the PR.

ivandrofly and others added 7 commits July 18, 2026 11:15
The allBoldItalic branch was a copy-paste of allUnderlineBoldItalic,
emitting ^U underline markers around bold-italic lines.
Before: <b><i>Hi</i></b> -> ^U^B^I|^I^B^U
After:  <b><i>Hi</i></b> -> ^B^I|^I^B

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Turkish branch removed/inserted at the colon position j instead of
the letter position k, corrupting the colon.
Before: "Ali: iyi" -> "Aliİ iyi"
After:  "Ali: iyi" -> "Ali: İyi"

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The \p{L}I\p{Ll} match guarantees a lowercase char at Index+2, but the
guard required Index < Length-3, so a match ending at the last position
read after='\0' and fell through to the lowercase-'l' replacement.
Before: "You HIt." -> "You Hlt."
After:  "You HIt." -> "You Hit."

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Encoding.Unicode is UTF-16LE, whose BOM is 0xff 0xfe (as ReadAllTextShared
already checks). The reversed 0xfe 0xff test never matched, leaving the
U+FEFF BOM in the first line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stripped closing tags were appended to post, so nested tags came out
reversed. Prepend to preserve order.
Before: <font color="white"><i>Hello</i></font> -> ...Hello</font></i>
After:  <font color="white"><i>Hello</i></font> -> ...Hello</i></font>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hebrew closing-italic case skipped only 2 chars after '<' instead of
3 ('/','i','>'), so the '>' leaked into the output. Mirror the Chinese
branch, which correctly uses skipCount = 3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin SplitEndTags suffix order, the FixUppercaseIInsideWords end-of-text
consonant check, the UTF-16LE BOM strip in ReadAllLinesShared, and the
DvdStudioPro bold+italic separator (encode + round-trip, no stray ^U).
All verified to fail against the pre-fix code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@niksedk
niksedk force-pushed the fix/text-corruption-fixes branch from bbe3ce8 to 80e513c Compare July 18, 2026 09:17
@niksedk niksedk changed the title Fix seven silent text-corruption bugs in libse Fix six silent text-corruption bugs in libse Jul 18, 2026
@niksedk

niksedk commented Jul 18, 2026

Copy link
Copy Markdown
Member

Thanks — six of the seven fixes verified cleanly and are merged, with regression tests added for each (SplitEndTags order, FixUppercaseIInsideWords end-of-text, UTF-16LE BOM, DvdStudioPro encode + round-trip, Turkish colon).

The Cavena890 Chinese index++ commit was dropped: the Chinese write path writes one byte per character (encoding.GetBytes(new[] { current })[0] takes only the high byte of the UTF-16BE pair) while the read side decodes the text field as UTF-16BE byte pairs (even-length aligned). So the path can't round-trip any Chinese text with or without the marker fix, and inserting a lone 0x88/0x98 byte actually shifts the pair alignment on read. Making Chinese writing work needs a real fix (two bytes per char + markers in a form the UTF-16BE decode maps back) — worth its own PR.

🤖 Generated with Claude Code

@niksedk
niksedk merged commit c6c6aae into SubtitleEdit:main Jul 18, 2026
1 of 2 checks passed
@ivandrofly
ivandrofly deleted the fix/text-corruption-fixes branch July 18, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants