Skip to content

fix(vm): tag binary-op result with the wider operand type (Susie's outfit crash)#360

Open
Ananim353 wants to merge 1 commit into
ButterscotchRunner:mainfrom
Ananim353:fix/binary-op-result-stacktype
Open

fix(vm): tag binary-op result with the wider operand type (Susie's outfit crash)#360
Ananim353 wants to merge 1 commit into
ButterscotchRunner:mainfrom
Ananim353:fix/binary-op-result-stacktype

Conversation

@Ananim353

Copy link
Copy Markdown

Problem

Arithmetic and bitwise handlers tag the result slot's gmlStackType with
instrType2(instr) (OP_ADD/SUB/MUL fast+slow, handleDiv/Rem/Mod, the bitwise
macro). A numeric binary op's result should take the wider of the two
operand types. When type1 is wider than type2 — e.g. add.v.i (Variable +
int) — the result is mis-tagged as the narrow type (INT32, 4 bytes) instead of
Variable (16 bytes). Since gmlStackType drives the byte-count model in
bytesToSlotCount (used by the Dup swap variant and BC17 BREAK
sub-opcodes), the mismatch makes the byte walk overshoot a slot boundary and
abort at require(remaining == 0).

Reproduce

DELTARUNE Chapter 5, Susie's outfit ("thrashfit") menu —
obj_ch5_LW07_thrashfit_menu.show_menu calls
option.init(id, label, names, flag, 170 + ja_offset). The last argument
170 + ja_offset compiles to add.v.i with no following Conv and flows
straight into the dup.v swap that arranges the 5-argument method call. The
swap expects 5 * sizeof(Variable) = 80 bytes; the mis-tagged INT32 argument
makes the real slots sum to 4 + 16*4 = 68remaining = -4 → abort. The
preceding 4-argument init (all Variables, 64 = 4*16) is fine, so it aborts
precisely on the first right-column option.

Fix

static uint8_t binaryResultType(uint32_t instr) {
    uint8_t t1 = instrType1(instr), t2 = instrType2(instr);
    return (gmlTypeNativeSize(t1) > gmlTypeNativeSize(t2)) ? t1 : t2;
}

Used at every binary-op result-tag site. It differs from instrType2 only
when type1 is strictly wider than type2 (the buggy case), so it cannot
regress working code. gmlStackType is read only by bytesToSlotCount, so
widening the tag never changes a value's runtime interpretation — only the
byte-count model.

Relation to #340

Fixes the same crash as #340 ("Susie's outfit"), but at the root cause — the
mis-tagged operand — instead of changing the Dup swap decoding. Because it
leaves the Dup/bytesToSlotCount byte-count model untouched, it does not
affect legitimate mixed-width Dup-swaps, so it should avoid the save-menu /
save-deletion regression noted on #340.

Verification

  • Builds clean (Release), no new warnings.
  • Passes the headless tests (object-index-iteration, struct-get-names,
    simple-structs-methods-and-dynamic-methods).
  • Verified on a downstream PSP port: the Chapter 5 costume menu, which
    previously aborted at require(remaining == 0), now opens correctly.

The arithmetic and bitwise handlers tagged the result slot's gmlStackType
with instrType2(instr). A numeric binary op's result should take the WIDER
of the two operand types; when type1 is wider than type2 (e.g. add.v.i,
Variable + int) the result was mis-tagged as the narrow type (INT32, 4B)
instead of Variable (16B). gmlStackType drives the byte-count model in
bytesToSlotCount (used by the Dup swap variant and BC17 BREAK sub-opcodes),
so the mismatch makes the byte walk overshoot a slot boundary and abort at
require(remaining == 0).

Reproduces in DELTARUNE Chapter 5, Susie's outfit ("thrashfit") menu:
option.init(id, label, names, flag, 170 + ja_offset) compiles the last
argument to add.v.i with no following Conv, feeding a .v Dup-swap arg list;
the mis-tagged INT32 slot makes the 5 args sum to 68 bytes instead of 80.

Introduce binaryResultType() and use it at every binary-op result-tag site.
It differs from instrType2 only when type1 is strictly wider than type2 (the
buggy case), so it cannot regress code that currently works. gmlStackType is
read only by bytesToSlotCount, so widening the tag never changes a value's
runtime interpretation.
@cobaltgit
cobaltgit requested a review from Fancy2209 July 25, 2026 16:41
@cobaltgit

Copy link
Copy Markdown
Collaborator

Can confirm this fixes the crash

@Fancy2209

Copy link
Copy Markdown
Collaborator

How did you find this was the issue?

@Ananim353

Copy link
Copy Markdown
Author

How did you find this was the issue?

To be honest, I spent several hours creating this fix using AI, so the answer below comes from it.

On a PSP port it was a silent abort() from require(remaining == 0) in
bytesToSlotCount, not an exception. Since that require is only reachable from the
Dup swap handler, I disassembled the Ch5 outfit menu's show_menu and found
option.init(id, label, names, flag, 170 + ja_offset): the last arg compiles to
add.v.i with no Conv and feeds a .v dup-swap that sizes its 5 args as
Variables (5*16 = 80). OP_ADD tags the result via instrType2i
(INT32, 4B), so the slots sum to 4 + 16*4 = 68, remaining never hits 0, and it
aborts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants