New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes#11
Open
joaopauloschuler wants to merge 46 commits into
Open
New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes#11joaopauloschuler wants to merge 46 commits into
joaopauloschuler wants to merge 46 commits into
Conversation
Introduce a node-level loop-invariant code motion pass over tfornode and
twhilerepeatnode bodies. It hoists side-effect-free, exception-free,
loop-invariant numeric/pointer subexpressions (constants and non-aliased
local/parameter reads combined with +, - and *) into the loop preheader as
write-once temps, so they evaluate once instead of on every iteration --
orthogonal to strength reduction, which rewrites the index recurrence rather
than lifting invariants out of it.
Soundness policy (a wrong hoist is a miscompile, so this is deliberately
conservative):
* invariance is proven via the DFA def-set of the whole loop node, which for
a for-loop includes the counter, so anything assigned in the loop -- the
counter included -- disqualifies the expression;
* only pure, non-trapping trees are hoisted (no calls, pointer derefs, array
indexing, div/mod, range/overflow-checked arithmetic, volatile/threadvars,
property/field access, or managed-type operations such as string "+"),
which makes preheader evaluation safe even for a zero-trip loop;
* aliasing is punted on: any addr-taken variable is treated as non-invariant
and no pointer/field/index read is ever hoisted, so a store through a
pointer in the loop cannot invalidate a hoist.
Gated by the new cs_opt_loopmotion switch (-OoLICM / {$OPTIMIZATION LICM}),
enabled by default at -O4 via genericlevel4optimizerswitches. Runs in psub
inside the DFA block, after induction-variable strength reduction, skipping
procedures that contain labels (as strength reduction does).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
testtool cases that fail if LICM miscompiles: an invariant multiply in a nested loop, a while-loop invariant, an invariant float subexpression, a zero-trip loop (hoist must stay harmless), a var modified in the loop (not invariant), pointer-aliased writes to an addr-taken var (not invariant), a div-by-a-zero-variable in a zero-trip loop (a trap must never be speculated into the preheader), and an invariant string "+" concat (a managed op must never be hoisted as arithmetic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port gcc's -funswitch-loops to FPC as a node-level pass over tfornode and
twhilerepeatnode bodies. When a conditional inside a loop tests a loop-invariant
expression, the test is evaluated once into a preheader temp and the loop is
cloned into a then-variant and an else-variant, so each cloned body is
branch-free for that test:
for i:=... do temp:=<cond>;
if <cond> then --> if temp then
A for i:=... do A { branch-free }
else else
B; for i:=... do B; { branch-free }
This is the neural-api kernel shape: an element loop that branches on a per-call
invariant flag (use-bias / apply-activation / stride / padding), paying a
compare+branch per element and blocking a uniform, vectorizable body.
Soundness policy (a wrong unswitch is a miscompile, so this is deliberately
conservative and reuses the LICM analysis):
* the condition must be pure, non-trapping and loop-invariant per the DFA
def-set of the whole loop node -- proven with the LICM helper factored out
as licm_is_pure_invariant, extended only with relational (=,<>,<,<=,>,>=)
and boolean-not wrappers, which likewise cannot trap or have side effects;
calls, derefs, div/mod, checked arithmetic and addr-taken/volatile/threadvar
reads are all rejected, so nothing that could trap is ever speculated;
* because the condition is pure and non-trapping, evaluating it once in the
preheader is safe even for a zero-trip loop;
* only one if is unswitched per loop (no cascading) and only when the body
fits a node-weight budget (node_count_weighted <= 50), so cloning cannot
blow up code size;
* nested loops are unswitched innermost-first (postorder), matching LICM;
* procedures containing labels are skipped at the call site, as strength
reduction and LICM do, so goto never crosses a clone boundary; break/exit/
continue inside a branch stay correct because the two clones are mutually
exclusive and each is its own loop.
The target if is marked by pointing its condition at the (unique) preheader
temp; after node_getcopy each clone carries an if testing that temp, which lets
each clone be specialised to its then/else branch unambiguously.
Gated by the new cs_opt_loopunswitch switch (-OoLOOPUNSWITCH /
{$OPTIMIZATION LOOPUNSWITCH}), enabled by default at -O4 via
genericlevel4optimizerswitches. Runs in psub inside the DFA block, immediately
before LICM with a DFA refresh in between, so the branch-free clones it produces
are exposed to invariant hoisting.
Tests under unleashed/tests/testfiles/optunswitch cover the condition-true and
condition-false paths, a zero-trip loop (preheader eval must stay safe), a
condition that would trap if speculated (must not unswitch), a non-invariant
condition and an addr-taken/aliased condition (must not unswitch), nested loops
(inner unswitched), break inside a branch, a while-loop, a relational condition,
and a body over the size cap (must not unswitch). Runtime trip counts plus
NOAUTOINLINE keep the loops from being fully unrolled so unswitching is the
transform actually exercised.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Recognize the canonical scalar clear-lowest-set-bit population-count loop
while x <> 0 do begin inc(c); x := x and (x - 1) end;
and rewrite it to
inc(c, PopCnt(x)); x := 0;
so the backend emits POPCNT under the CPUID feature gate (e.g. -CpCOREAVX2)
and the fpc_popcnt_* RTL routine otherwise, replacing a multi-instruction
scalar loop with a single instruction in the hot paths of bit-heavy code
(core.bloom / core.hyperloglog / core.roaring / core.bits).
Pass OptimizeBitIdiom in optloop.pas, driven from the psub.pas DFA block
next to OptimizeLICM / OptimizeLoopUnswitch, gated by the new switch
cs_opt_bitidiom (-OoBITIDIOM) which genericlevel4optimizerswitches enables
at -O4. Switch plumbing mirrors the two loop passes: globtype.pas enum +
OptimizerSwitchStr, x86_64/cpuinfo.pas supported_optimizerswitches, and
ppudump.pp.
Recognized shapes (a false positive is a miscompile, so matching is strict):
* the two body statements in either order;
* the counter as inc(c) or as c := c + 1 (inc() is lowered to the latter by
firstpass before this pass runs);
* the unsigned while x > 0 guard (rejected for signed x);
* x and c distinct simple, non-aliased, non-volatile, non-threadvar
local vars or value params, ordinal, exactly 32 or 64 bits;
* the counter step exactly +1; the body exactly the two statements;
* only bit-preserving same-size ordinal reinterpret typeconvs are looked
through, so a narrowing while Word(x) <> 0 is never misread;
* signed x is reinterpreted to the same-width unsigned type before PopCnt,
so the count matches the loop for negative/high-bit values.
The -O2 while -> "if cond then do..while cond" simplification turns the loop
into a test-at-end form by the time this pass sees it; for that form the
rewrite is wrapped in a zero-trip if <cond> then ... guard so x=0 stays a
no-op (PopCnt(0)=0). repeat..until is rejected: it runs the body once for
x=0, which PopCnt would miscount.
Tests under unleashed/tests/testfiles/optbitidiom/: positive rewrites (dword,
qword, reversed body, c:=c+1, x>0, signed 32/64) and must-not-fire negatives
(step +2, extra statement, x shr 1 bit-length, repeat..until), verified by
behaviour and by assembly (popcnt / fpc_popcnt present vs absent).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New optimizer switch cs_opt_rangecheckelim / -OoRANGEELIM (in
genericlevel4optimizerswitches, so on by default at -O4). Pass
OptimizeRangeElim in optloop.pas, driven from the DFA block in psub.pas
after OptimizeBitIdiom and skipped for procedures containing labels.
When range checking is on (-Cr), the per-access array bounds check that
-Cr injects in front of a[i] inside a counted for-loop is removed when
the loop counter i is provably always a valid index. The suppression
reuses FPC's own per-node range-check switch: for a dynamic array the
check (fpc_dynarray_rangecheck) is gated on the vecn's localswitches, and
for a static array it lives in the index->subrange typeconv child, so the
pass clears cs_check_range on both the vecn and its index child -- exactly
the primitive tvecnode.gen_array_rangecheck already uses in nmem.pas.
Two deliberately conservative patterns qualify (a wrongly removed check is
a memory-safety miscompile):
A. static array[Lo..Hi] indexed by the counter whose *constant* loop
bounds lie inside [Lo..Hi];
B. dynamic array a (a simple non-aliased local/value-param loadn, not
address-taken, DFA-proven not reassigned in the body) indexed by the
counter when the loop bound is literally high(a) or length(a)-1 and
the other bound is a constant >= 0.
The counter must be a simple non-aliased local/value-param variable not
written in the loop body (DFA def-set). Everything else keeps its check:
offset indices (a[i+1]), a manually computed index, a different array, a
plain length(a) off-by-one bound, a global counter, or an array
SetLength'd smaller inside the loop. Zero-trip safe (high(nil) = -1).
Plumbing mirrors the OoBITIDIOM pass: globtype.pas (enum + OptimizerSwitchStr
'RANGEELIM' + genericlevel4), x86_64/cpuinfo.pas supported_optimizerswitches,
ppudump.pp. Tests under unleashed/tests/testfiles/optvrp/ cover in-bounds
static/dynamic/backward/length-1 loops (check-free, correct output),
zero-trip nil, and five must-still-raise cases (i+1, off-by-one, SetLength
shrink, manual index, runtime-variable bound). Assembly diff confirms the
qualifying inner loop drops all fpc_rangeerror/fpc_dynarray_rangecheck;
a hot c[i]:=a[i]*b[i]+c[i] loop built -Cr -O4 is ~2.9x faster with the
pass on, byte-identical output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrites the canonical single-precision element-wise loop for i := lo to hi do a[i] := b[i] <op> c[i] (op in + - *) over simple non-aliased dynamic arrays of single into a 128-bit SSE (or AVX-encoded, when available) packed main loop processing 4 lanes per iteration plus a scalar remainder loop. Loop control stays as ordinary nodes; only the 4-lane body is a dedicated backend node (tvectoropnode, x86 override emitting movups + addps/subps/mulps). The recognizer is strict so a miscompile cannot slip through: signed non-aliased local counter proven unmodified by DFA, exactly one plain assignment of the recognized shape, same index on every access (so element-wise semantics make it alias-safe without runtime guards), range/overflow checking disables the transform, and the FP result is bit-identical to scalar (no reassociation, no fast-math gate needed). Opt-in via -O4 -OoVECTORIZE for now; promoting it into the -O4 default set is a follow-up once it has soaked. Tests: unleashed/tests/testfiles/optvect/ — verified bit-exact against scalar recomputation for all tail residues, aliasing (a:=b shared block), negatives/special values (NaN/Inf/-0.0), -Cr fallback, and the AVX encoding; full 804-test suite green (3 pre-existing environment failures unrelated to this change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed `x mod c` with c not a power of two previously fell through to a hardware idiv. Add a magic-number lowering in tx86moddivnode.pass_generate_code (x86/nx86mat.pas): reuse the existing signed-division magic sequence to compute the quotient q := x div c, then derive the remainder as x - q*c, avoiding idiv. Gated identically to the sibling const paths (right is ordconstn, signed, sizes 4 and 8) and placed after the power-of-two signed-mod branch so that fast path is unaffected. Bit-exactness verified against hardware/reference semantics for negative dividends (truncation toward zero), Low(LongInt)/Low(Int64), divisors c and -c (3, 7, 10, 1000003, and Low(Int64) mod -1 which traps idiv but the magic/pow2 path returns 0), across longint and int64. New tests under unleashed/tests/testfiles/optmodconst/. Full regression suite green (only the known pre-existing failures remain). Assembly check: signed `mod 10`/`mod 1000003` emit imul+shifts, no idiv. Micro-benchmark (-O3, tight x mod 10 loop, 3000 * 2M iters): magic 15.46 s vs idiv 25.54 s (~1.65x faster), identical results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend OptimizeBitIdiom with two scalar bit-scan loop shapes, lowered to
the Bsf/Bsr intrinsics (TZCNT/BSF, LZCNT/BSR under the CPU feature gate):
* count-trailing-zeros: if x<>0 then while (x and 1)=0 do
begin inc(c); x:=x shr 1 end
-> inc(c, Bsf(x)); x := x shr Bsf(x)
The scalar loop is infinite for x=0, so this is only sound when a
dominating nonzero test proves x<>0. The pass therefore anchors on the
enclosing if x<>0 (or unsigned x>0), keeps that guard verbatim and only
rewrites its body; loops without such a structural proof are left alone.
* highest-set-bit / floor-log2: while x>1 do begin inc(c); x:=x shr 1 end
-> if x<>0 then begin inc(c, Bsr(x)); x := 1 end
This loop is total (zero-trip for x in 0 and 1); the emitted if x<>0 only
avoids the undefined Bsr(0) and reproduces the x=0 no-op. Unsigned only.
Both shapes are recognized in the raw test-at-begin while form and in the
-O2 do-while-in-if form (shared bitidiom_scan_loop_parts helper, which
verifies the do-while entry guard equals the loop condition). Restricted to
32/64-bit ordinal locals/value-params, exact +1 counter step, exactly the
two-statement body -- the same strict policy as the population-count matcher
it reuses. 8/16-bit operands are integer-promoted and fall through to the
scalar loop unchanged.
Tests under unleashed/tests/testfiles/optbitscan/ verify bit-exact identity
with a scalar reference across all single-bit values, dense sweeps, High/Low
and the guarded x=0 no-op for 8/16/32/64-bit (and signed 32-bit tzcnt), and
a must-not-fire set whose results would differ if the pass misfired.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er offsets
Induction-variable strength reduction previously only lowered a bare
x := i*stride in a for-loop to an additive accumulator; the same multiply
buried in an array index or pointer offset (g[i*7], p[i*stride],
p[ch*size + row*width + col]) still emitted a per-iteration imul, because
FPC wraps the counter load in an implicit value-preserving conversion for
index/offset arithmetic, so taddnode(n).left was a typeconvn, not the bare
loadn the MUL case matched.
dostrengthreductiontest's MUL case now recognises the counter through a
chain of value-preserving integer conversions (is_reducible_loopvar /
is_value_preserving_int_conv: strict integer widening, not signed->unsigned).
Soundness gates mirror the existing pass and are deliberately conservative:
* the multiply is skipped when cs_check_overflow/cs_check_range is active
on the muln node (a checked multiply may trap and must run every
iteration); this also preserves the historic behaviour that -Cr/-Co
disabled the reduction (previously by accident, via the check-emitting
typeconvs breaking the loadn match);
* a conversion carrying a range/overflow check is never seen through;
* only plain reads of the counter (no nf_write/nf_modify) qualify;
* the multiplier must be loop-invariant (unchanged: is_loop_invariant).
Nested conv-like offsets reduce level-by-level: each loop's own
counter*invariant term (ch*size, row*width) is reduced when that loop is
processed; the innermost +col add stays. Range checks on the index are
unaffected because the reduction is gated off under -Cr and, when it does
fire, feeds the vecn the identical index value via the accumulator temp.
New tests under unleashed/tests/testfiles/optstrred/ verify bit-exact
results vs a naive recompute for constant/runtime strides, the multi-term
flat offset in a nested loop, backward loops, boundary trip counts (0/1/many),
a non-matching variant-stride case, and -Cr correctness. Full regression
suite green (minus the known pre-existing failures). On a 32x64x64
conv-like flat-offset nest the inner loop drops from 6 imuls to 0 and runs
~35% faster (1.9s vs 3.0s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Report, per hot for-loop, whether -O4 -OoVECTORIZE autovectorized it and, when it did not, the concrete reason the recognizer bailed. This is a maintainer feedback signal so loops that fall back to scalar code no longer have to be found by reading generated assembly. It measures/reports only: codegen is unchanged (verified by a byte-identical .s diff with/without the patch). Mechanism: - Two new codegenerator notes in msg/errore.msg: cg_n_loop_vectorized (06065) and cg_n_loop_not_vectorized (06066, "$1" = reason). Regenerated msgidx.inc and msgtxt.inc via utils/msg2inc (both are .gitignore'd build artifacts, so they are rebuilt from errore.msg at compile time and not committed). - Instrumented tvectorizecontext.processloop: the recognizer body moved into a nested vectorize_reason function whose every early exit now carries a human-readable reason string (descending loop, non-unit step, aliasing, non-single element type, multiple statements in body, checked code -Cr/-Co, non-counter/offset index, unsupported operator, counter modified in body, ...). On failure it emits 06066 at the for-loop position; on success 06065. - vect_single_dynarray_elem is reimplemented on top of a new vect_elem_reason helper so per-operand array-shape failures (destination/first/second source) get precise reasons. Zero-cost and gating: OptimizeVectorize only runs when cs_opt_vectorize is set (psub.pas), so normal builds never build reason strings or emit anything. The notes follow standard FPC note gating -- shown at default/-vn verbosity, suppressed with -vn-. Verified manually: - vectorizable single a[i]:=b[i]+c[i] in a proc prints "Loop autovectorized (SSE, 4-wide packed single)" at the loop line; - double elements -> "array element type is not single-precision float"; two-statement body -> "loop body is empty or has multiple statements"; b[i+1] index -> "array index is not a plain variable read"; descending -> "descending (downto) loop"; -Cr -> "range/overflow checking is enabled (-Cr/-Co)"; - without -OoVECTORIZE nothing new is printed; - .s output for a sample is byte-identical with and without the patch. Two run+compile tests added under unleashed/tests/testfiles/optvect/ exercising the diagnostic active (-vn) on vectorized and several non-vectorized loops, asserting bit-exact scalar-equivalent results. Full suite green (814 pass; only the documented pre-existing failures remain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the conservative SSE/AVX autovectorizer (-O4 -OoVECTORIZE) beyond the
array-array a[i]:=b[i] op c[i] shape to two more single-precision element-wise
kernels neural-api's TNNetVolume uses constantly:
* scalar broadcast: a[i] := b[i] op s and a[i] := s op b[i]
where s is a loop-invariant single (a simple non-aliased local/value-param
the single-statement body never writes, or a constant literal). The scalar
is splatted ONCE before the vector loop (movss/vmovss + shufps/vshufps into
a 16-byte memory slot, hoisted by a new vok_broadcast tvectoropnode kind),
then a packed add/sub/mul per iteration. Non-commutative subtraction keeps
s - b[i] per lane via the scalarleft flag.
* plain copy: a[i] := b[i] -> a packed movups load+store pair with a scalar
tail.
tvectoropnode grows a `kind` (vok_arr_arr/arr_scalar/copy/broadcast) and a
`scalarleft` field; the x86 pass_generate_code switches on kind. Soundness gates
mirror the array-array shape exactly: dynamic arrays of single, plain-counter
index on every access, single-precision arithmetic only (is_single guard on the
RHS and the scalar operand, so a double/int scalar that would widen the scalar
loop is rejected), -Cr/-Co disables, and the invariant scalar must be a simple
non-global, non-address-taken, non-volatile local/param or a constant. FP
results stay bit-identical (same per-lane op and order, the broadcast puts the
identical bit pattern of s in every lane; NaN/Inf/-0.0 propagate). The
-OoVECTORIZE diagnostic (06065/06066) reason strings are extended for the new
near-misses.
optdfa: register vectoropn in the CheckAndWarn uninitialized-variable walk
(MaybeSearchIn) -- reachable now that a live scalar is read by the body node.
New optvect tests: scalar broadcast across all ops incl. non-commutative and
constant operands (trip counts 0..17, 100, 1000), plain copy, special values
(NaN/Inf/-0.0, bit-compared), self-alias, and global/address-taken rejection.
Full suite green (minus known pre-existing failures). Micro-benchmark of a
memory-bandwidth-bound b[i]*s kernel: ~1.13x.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stock -Oodeadstore pass only removes a redundant store whose LHS is a
whole-variable loadn (redundancy decided from DFA whole-variable liveness,
which is too coarse to reason about an individual field or element). Add a
second, memory-location-precise straight-line pass that kills a store to a
record field or a constant-index static-array element when a LATER store
overwrites the exact same location before anything can observe it.
The extended pass keeps a per-run "pending" set of stores to compile-time
known slots (base variable + constant field/element path) and removes an
earlier store only when a later store to the structurally-equal location
(tnode.isequal: same base sym, same field vs, same constant index) is reached
with no intervening observer. It is deliberately over-conservative:
* base must be a non-address-taken local / value-or-const parameter /
non-different-scope static var of the current routine (same predicate as
the stock pass), so two distinct bases can never overlap;
* the location path may only contain record-field subscripts and constant
ordinal indexes into genuine non-packed non-special static arrays -- any
pointer-backed access (p^, dynamic/open array) is rejected because it
could alias a different base variable (e.g. f(x,x));
* managed/ref-counted slot types are skipped (finalization side effects);
* the RHS must be free of exception/range/overflow side effects and contain
no call, pointer deref or asm node;
* any read of the base variable, any call, deref, asm, compound assignment
or control-flow construct between two stores keeps the earlier store.
The function-result variable is intentionally not covered: it is passed by
reference and can alias a var/const parameter (a := f(a)), which base-only
reasoning cannot see -- the stock whole-var pass excludes it for the same
reason.
Level wiring: left opt-in under -Oodeadstore (not promoted into -O4). A forced
-O4/-Oodeadstore sweep showed the pre-existing stock pass already miscompiles
several mode-unleashed constructs (statement-expressions, inline vars,
function references) that were never exercised because the switch is in no -O
level; promoting it would regress those. A -O3 -Oodeadstore sweep over the
full testfiles corpus is byte-for-byte identical in pass/fail with the
extension on vs off, i.e. this change introduces no new failures.
Adds unleashed/tests/testfiles/dead_store_ext/ covering record-field and
array-element removal, the read/call/address-taken/managed safety matrix, and
soundness under -Cr -Co.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add CallFrameRet2Jmp, a -O4 post-peephole that converts a sibling tail call
made from a routine that owns a stack frame into a jump. When the caller has
its own frame, the teardown sits between the call and the ret
call X
L: leaq N(%rsp),%rsp ; stack release
popq %rbx ; optional callee-saved restores
ret
so the existing CallRet2Jmp / Lea/PushCallRet variants no longer match. The
new pass hoists a copy of the teardown above the call and rewrites the call to
a jump, leaving the original teardown+ret in place (label L may be the merge
point of other, non-call, exit paths such as base cases):
leaq N(%rsp),%rsp
popq %rbx
jmp X
Covered subset (chosen for provable peephole-level soundness):
* teardown = a stack release (leaq N(%rsp),%rsp or addq $N,%rsp, N>0)
and/or pops of callee-saved integer registers only. Callee-saved regs
are never argument registers, so hoisting the pops cannot clobber an
outgoing argument of X, and the teardown restores rsp exactly to routine
entry, so the jump target is entered with ABI-mandated alignment
regardless of target_info.stackalign.
* only labels/directives may sit between the call and the teardown, so the
callee's result is returned unchanged (a genuine tail call). A result
routed through a callee-saved register (mov %rax,%rbx after the call) is
a separate result-forwarding problem and is deliberately not handled.
Soundness gates (CurrentProcAllowsSiblingTailFrameReuse): reject routines with
any address-taken local/parameter (a pointer into the released frame could be
an argument), routines whose frame is captured by a nested routine
(parentfpstruct assigned), assembler routines, implicit-finally / exception
frames (pi_uses_exceptions, pi_needs_implicit_finally, pi_has_implicit_finally)
and dynamic stack allocation (pi_has_stack_allocs).
Tests (unleashed/tests/testfiles/optsibcall): deep void and callee-saved-pop
mutual recursion that overflow the stack at -O2 but run in O(1) stack at -O4;
negative behavioural cases (pointer-to-local, try/finally, managed local) that
must keep the call and stay correct. Full suite green (minus known
pre-existing failures). Isolated micro-benchmark (same -O4, peephole off vs
on) on a deep mutual tail-call chain: ~22.9 s -> ~11.8 s (~1.9x).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fthread-jumps / VRP-based jump threading to the FPC node
tree: when a dominating branch (or a value-range fact) already decides a
predicate, a nested if that re-tests it folds straight to the taken arm,
deleting the redundant compare/branch and the dead arm. Unlike
if-conversion or GVN-PRE this restructures control flow itself.
Two fact kinds, both strictly gated (a wrong-way implication is a
miscompile):
* comparison facts V <op> c (=,<>,<,<=,>,>=; c ordinal const; V a
simple non-aliased, non-address-taken, non-volatile, non-threadvar
local/value-param). A nested comparison on the same V and same
signedness is decided by integer implication over the unbounded
number line (jt_all_satisfy): S(fact) subset of Q -> true, disjoint
-> false, else undecided. Ignoring the type's domain bounds can only
lose folds, never create an unsound one.
* identical-predicate facts: any side-effect-free, call-free,
memory-free condition over constants, simple-var loads and pure
operators folds a tnode.isequal re-test.
A fact is asserted for a branch only when its variable(s) are provably
unassigned in that whole branch (jt_writes_var), so it stays invariant
throughout -- a re-test anywhere in the region, even inside a nested
loop, is decidable. Address-not-taken rules out by-ref/pointer writes,
so an ordinary call on the path cannot touch the variable and does not
block the fold. Facts inherited from an enclosing dominating if are
carried down unchanged, letting chained if/elsif ladders fold later
re-tests of earlier decisions.
Folding removes code (no duplication), so there is no code-growth budget
to manage. Wired as cs_opt_jumpthread / 'JUMPTHREAD', added to
genericlevel4optimizerswitches (-O4) and x86_64 supported switches; the
psub call refreshes DFA since the CFG changes; skips procedures with
labels.
Tests (unleashed/tests/testfiles/jumpthread): jt_exhaustive cross-checks
432 signed + 432 unsigned outer/inner comparison-pair combinations over
the full x domain against an unfoldable runtime-op reference (catches
any wrong-way implication); jt_gates proves an intervening assignment,
an address-taken store, a global and a threadvar all block the fold
while a call between does not; jt_evidence and jt_ladder use
%CHECKBIN_LACKS to show the dead arm's string literal disappears.
Full suite green (minus the 4 known pre-existing failures), including a
forced-on sweep across the whole corpus. Chained-decision microbench
~25-30% faster with threading on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ies (-Oodeadstore) The dead-store switch (-Oodeadstore) gates a call to opttree.normalize() in psub, which only runs at -O3+ (inside the cs_opt_nodedfa block). normalize's searchblock walked a statement's whole expression subtree with a single fixed hoist target (the enclosing statement), but foreachnodestatic descends through nested statement lists too -- so a statement/if/match-expression block buried in a loop or if body was hoisted to BEFORE the loop. The block-expression was then evaluated once with a stale counter and its temp reused every iteration, so e.g. `for n... var s := if n=0 then 'zero' else 'other'` produced 'other' for n=0. This mis-hoist caused ~a dozen mode-unleashed miscompiles at -O3 -Oodeadstore (statement_expr/*, match, inline_vars inferred-array-string, array_size_shortcut). Fixes: - searchblock now STOPS at a nested statementn (does not hoist its block-expressions past its own enclosing statement); searchstatements grows a second phase that recurses into nested statement lists (loop/if/case/try bodies) and normalizes each with its own correct insertion point. - The "move the whole block out of the expression" branch replaced a statement slot with a raw block node and re-firstpassed a managed init/cleanup temp; for interface / function-reference / some generic-intrinsic block-expressions this dereferenced freed/mis-typed nodes and crashed the compiler (funcref_*, type_intrinsic_generic_arg_13). None of the constructs this pass must normalize need that branch (they yield a temp-ref or constant), so it now declares the proc not-normalizable (normalize returns false, DSE is skipped for it, the block-expression is left exactly as with the switch off) instead of crashing. - guard the post-phase-1 sibling walk against the slot no longer being a statementn. Result: at -O3 -Oodeadstore the suite has the same failure set as -O3 alone (816/22; the remainder are pre-existing -O3 loop bugs), so -Oodeadstore adds no failures and is now promotable. Adds regression tests under unleashed/tests/testfiles/dead_store_normalize/ (all fail on the old compiler). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…itives + case exhaustiveness)
A stage-2 self-compile at -O4 (our stage-1 compiler recompiling the compiler
with OPT=-O4) failed under -Sew (warnings-as-errors) on a series of
warning-as-errors, none caused by the new optimizer passes -- they are DFA
uninitialized-variable false positives and case-exhaustiveness warnings in
fork-added (and a couple of stock) routines that had simply never been
self-compiled at -O2+/-O4 (the standard build bootstraps with system FPC 3.2.2,
which does not run DFA / emit these). Fixes, all behaviour-preserving:
- defutil.walk_field_path_bits: accumulate the running bit offset in a plain
local instead of directly in the "out" parameter (bit_total). Reading an out
param from inside an inlined nested procedure tripped a DFA "not initialized"
false positive; the local is written back to bit_total on success.
- optloop (vectorize / jumpthread / bitidiom passes):
* initialize the recognizer-result locals (counter, ctype, avec/bvec/cvec,
scalarnode, scalarleft, vecop, vshape) that the nested vectorize_reason fills
-- per-proc DFA cannot see a nested routine define parent locals;
* default-initialize the managed jt_pure_cond out-record before the call;
* add explicit "else ;" to the bitidiom node-type case and the jt_relkind /
jt_all_satisfy comparison/kind cases (case-exhaustiveness warning).
- pexpr / pstatmnt indexed-label parsing: initialize the label index (arrlabidx /
labidx) that is only assigned on the ordinal-constant path but always passed to
get_or_create_indexed_labelsym (ignored there for string labels).
- dbgdwarf flexible-array-member DWARF: initialize count_size and delta, which
are set and used only under correlated assigned(flexcount) guards that DFA
cannot relate.
With these, `make -C compiler ppcx64 PP=<stage1> OPT=-O4` compiles and links the
whole compiler cleanly, so an -O4 self-compile can serve as a self-host smoke
test. Full unleashed suite unchanged (835/7, only the documented pre-existing
failures).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uild for tests (task 273) Root-causes the three pre-existing bare-compiler unleashed failures (none are optimizer-related, none are /etc/fpc.cfg artifacts -- behaviour is identical with and without the installed config): - inline_vars_inferred_double_01 (FIXED): `var pi := 3.14` inferred the default best real type (Extended, SizeOf 10) instead of Double, so SizeOf(pi) <> SizeOf(Double). The array-literal inference path already normalizes float literals to Double (s64floattype) -- the scalar inline-var path did not. Promote a plain real-constant initializer of the default extended type to Double in inline_var_statement (guarded: not an explicit cast, only s80real/sc80real, only a realconstn), matching the array path and the documented intent. - multi_var_init_double_01 (documented, not a compiler bug): `x: Double = 3.14; if x <> 3.14`. The untyped real constant 3.14 is Extended, and comparing a Double variable against it is done at Extended precision, where Double(3.14) and Extended(3.14) differ (3.14 is inexact). The sibling multi_var_init_typed_const passes only because it uses 3.5, which is exact in both. This is stock FPC real-constant precision semantics; the test would need Double(3.14) or a tolerance. Not fixable without changing float comparison semantics globally. - inline_vars_inferred_array_string_widestring_01 (documented, not feasibly fixed): `[w, 'two', 'three']` with w: WideString. The array-constructor typecheck unifies the elements to their common string type (UnicodeString, since WideString loses to UnicodeString) and retypes the first element node in place *before* unleashed_infer_array_literal reads its "first element wins" kind, so the original WideString is unrecoverable from the post-typecheck tree (there is no typeconv wrapper to look through). The sibling unicode test passes only because UnicodeString is itself the unification winner. Fixing it would require capturing element types before the constructor unifies -- invasive and regression-prone for one niche case. Also documents in unleashed/tests/README.md the minimal fork-compiler build step for the `Rtti` unit (packages/rtl-objpas/src/inc/rtti.pp); with it on the unit path all three composable_records_rtti_flatten_* tests compile and pass (they were only failing on a missing unit path, not a compiler bug). Full suite 836/6 (was 835/7): inline_vars_inferred_double_01 now passes; the remaining failures are the two documented-above tests, the runner-artifact incfile test, and the three Rtti tests when run without the unit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-loop-distribute-patterns to FPC: recognizes a counted,
ascending, unit-stride for-loop whose *entire* body is one store walking a
contiguous array region and lowers the whole loop to the RTL block primitive
the C runtime already tunes per target, instead of a per-element scalar store:
for i:=lo to hi do a[i]:=0; -> FillChar(a[lo], n*sz, 0)
for i:=lo to hi do a[i]:=v; -> FillChar/FillWord/FillDWord/FillQWord (sz 1/2/4/8)
for i:=lo to hi do a[i]:=b[i]; -> Move(b[lo], a[lo], n*sz)
n = hi-lo+1, guarded by if lo<=hi (a for-loop is a no-op when lo>hi and the
byte count must not come from a negative difference); the count is widened to
SizeInt so n*sz cannot wrap.
Shapes recognized:
* zero-fill of any unmanaged element (ordinal/float/pointer/enum/record) via
FillChar over the byte span (constant 0/nil/+0.0; -0.0 is rejected);
* non-zero fill of an ordinal/pointer element of size 1/2/4/8 with a constant
or loop-invariant value, reinterpreted to the matching unsigned width so the
identical bytes the scalar store would write are emitted;
* element-wise copy between two dynamic arrays -> Move. Two dynamic-array
references can only fully alias (share the block at offset 0, after a:=b) or
be disjoint -- a shifted overlap is impossible -- so Move (memmove-safe)
reproduces the forward copy in both cases with no runtime guard.
The destination base may be any side-effect-free, counter-independent l-value
(e.g. Self.FData), so field-backed dynamic arrays like neural-api's
TNNetVolume.FData lower too, not just plain variables.
Shapes declined (compile exactly as before): downto / non-unit step, multi-
statement bodies, non-counter (offset) indices, managed element types, non-zero
float fills (a float->int cast would round, not reinterpret), element-type-
changing copies (double[i]:=single[i]), copies whose source/dest are static
arrays or pointers (a shifted overlap cannot be excluded), -Cr/-Co checked code,
inline candidates, and procs with labels. The counter is required signed 32/64-
bit, simple and non-aliased, and proven unmodified in the body by DFA; the
lowered block leaves the counter at hi (the value an executed ascending for-loop
leaves) inside the lo<=hi guard, matching scalar semantics.
New switch cs_opt_loopdistpat (-OoLOOPDISTPAT), node-level pass in optloop.pas
(postorder, so nested loops lower first), call site in psub before strength
reduction and the for->while lowering, added to genericlevel4optimizerswitches
so plain -O4 enables it. Per-loop diagnostics 06067/06068 (-vn) mirror the
VECTORIZE notes.
Benchmark (fantastica/nn-bench-style volume clear+copy over `array of single`,
best-of results, identical `acc` checksum on/off proving equal output):
N (singles) clear -O4 off -> on copy -O4 off -> on
64 185 ms -> 18 ms ~10x 150 ms -> 26 ms ~5.8x
4096 595 ms -> 46 ms ~13x 529 ms -> 41 ms ~13x
65536 806 ms -> 87 ms ~9x 645 ms -> 79 ms ~8x
1048576 460 ms -> 80 ms ~5.7x 492 ms ->253 ms ~1.9x
Large copies converge toward memory-bandwidth bound; clears and cache-resident
sizes see the biggest wins from the hand-tuned RTL primitives replacing FPC's
scalar element store loop.
Tests: unleashed/tests/testfiles/optdistpat/ -- fill (zero + every unmanaged
width), non-zero FillChar/Word/DWord/QWord, copy (ordinal/float/record, self-
copy, shared-block a:=b), static arrays incl. partial-range and declined static
copy, field-backed TNNetVolume-shaped clear+copy, post-loop counter value
(executed hi and non-executed unchanged), the -O4 default set, and a -vn
negatives file asserting each decline path stays correct. Full unleashed suite
836->844 pass / 6 fail (the 6 are the documented pre-existing failures), stage-2
-O4 self-compile clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port of gcc's -fpeel-loops (full-peel case) to the FPC node tree. When a
counted for-loop has a small compile-time-constant trip count, replace it
with that many straight-line copies of the body, folding the induction
variable to its per-iteration constant and deleting the counter, the loop
compare and the back-branch. Each peeled copy is then exposed to ordinary
constant folding/propagation (a[i] -> a[k] with k constant).
This is the deliberate -O4-only complement to the stock unroller
(unroll_loop in tfornode.pass_1): that heuristic already fully unrolls a
constant-count loop when the whole loop fits its unroll budget, but declines
the medium-body / low-trip kernels (3x3 convolution taps, per-channel Depth
1/3/4 loops) that sit inside hot outer loops where the loop control rivals
the body. LOOPPEEL picks those up: trip count <= 8, per-body weighted-node
cap 40, trip*body growth cap 160.
Soundness gates (a wrong peel is a miscompile):
* unit step, both bounds ordinal constants (exact trip count known);
* counter a simple non-aliased, non-address-taken, non-volatile local /
value param of ordinal type, DFA-proven unmodified in the body;
* body free of break/continue/goto/label/exit/raise;
* not TP/Mac mode, not an inline candidate, not -Os.
A statically empty loop (trip <= 0) is declined so the ordinary for-loop --
which leaves its counter unmodified when it never runs -- is what executes;
when the loop does run, the counter is left at its last taken value (the "to"
bound), matching normal for-loop post-conditions and keeping DFA consistent.
Wiring: cs_opt_looppeel switch (globtype.pas + ppudump table), added to
genericlevel4optimizerswitches; OptimizeLoopPeel in optloop.pas run in
psub.pas right after LOOPDISTPAT (still on for-nodes, before the for->while
lowering and strength reduction); notes cg_n_loop_peeled (06069) /
cg_n_loop_not_peeled (06070).
Tests (unleashed/tests/testfiles/optpeel, 6 files): ascending/descending
correctness across trip counts; counter-after value (last index when run,
untouched when statically empty); boundary trips (1, 8, 9-declined,
over-budget-declined); negatives that must decline (break, continue,
non-unit step, global counter) and stay correct; -O4-default firing; nested
peelable loops. Full suite 850 pass / 6 fail (the 6 known pre-existing);
-O4 stage-2 self-compile stays clean. Peeled kernel verified branch-free in
the emitted assembly (no back-branch or counter); representative hot-kernel
microbench ~3-7% faster (noisy).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… -O4
Port of gcc's -fsplit-loops to the FPC node tree. When the whole body of a
counted for-loop is a single conditional whose predicate compares the
induction variable against a loop-invariant bound m (if i<m then A else B and
its <=, >, >= / swapped-operand relatives), the truth value flips exactly once
as i sweeps the range, so the iteration space splits cleanly at the crossover:
for i := lo to hi do --> c := clamp(crossover, lo, hi+1);
if i < m then A else B for i := lo to c-1 do A; {branch-free}
for i := c to hi do B; {branch-free}
Both sub-loop bodies are branch-free: the per-iteration compare is gone, the
interior loop (the bulk of iterations -- e.g. the non-border rows of a padded
convolution) becomes a uniform kernel a later pass can vectorize, and only the
short border loop keeps the boundary work. Because the split runs before the
autovectorizer in psub, the exposed interior loop is offered to it directly.
Distinct from loop unswitching (whose condition is invariant in *both* operands
and never changes across iterations): here the IV side crosses the bound once.
Soundness gates (a wrong split is a miscompile):
* ascending unit-step loop; counter a simple non-aliased signed 32-bit local /
value param, DFA-proven unmodified in the body (32-bit so the widened int64
crossover math m+-1 / hi+1 cannot overflow);
* body is exactly one if whose condition is i <rel> m, rel in < <= > >=
(monotone in i; = and <> rejected), m an ordinal constant or a simple
non-aliased ordinal (<=32-bit) local / value param DFA-proven unmodified in
the body (evaluating it once == each iteration, cannot trap);
* neither branch contains break/continue/goto/label/exit/raise (a break in one
sub-loop would leave the other running);
* not TP/Mac, not inline candidate, not -Os, range/overflow checking off.
The crossover is clamped into [lo, hi+1] in an int64 domain and each sub-loop is
emitted under an `if lo<=c-1` / `if c<=hi` guard, which both skips a statically
empty sub-range and keeps the narrowing of c back to the 32-bit counter type in
range. The two loops tile [lo,hi] contiguously, so the counter is left with
exactly the value a single for-loop leaves (hi when it ran, unchanged if not).
Wiring: cs_opt_loopsplit switch (globtype.pas + ppudump table), added to
genericlevel4optimizerswitches; OptimizeLoopSplit in optloop.pas run in psub.pas
just before the vectorizer; notes cg_n_loop_split (06071) /
cg_n_loop_not_split (06072).
Tests (unleashed/tests/testfiles/optsplit, 5 files): exhaustive crossover sweep
(m below/inside/above the range, empty/short/long ranges) for all four relations
and the swapped operand order vs a scalar reference; counter-after value (hi when
run, untouched when empty); 8 declined shapes (=, <>, expression bound, fully
invariant condition, multi-statement body, break in a branch, downto, 64-bit
counter) each staying correct; -O4-default firing; the padded-stencil border
shape. Full suite 855 pass / 6 fail (the 6 known pre-existing); -O4 stage-2
self-compile stays clean. Interior loop verified branch-free in the emitted
assembly (the per-iteration i<m test is hoisted entirely into the prologue).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports the gcc/LLVM loop-fusion transform to FPC (the inverse of the already
landed -OoLOOPDISTPAT distribution): two ADJACENT counted for-loops over the
identical iteration space are merged into one loop whose body is the two
original bodies concatenated, so an intermediate result A writes and B
immediately re-reads stays in registers/cache for one pass instead of being
streamed out to DRAM by the first loop and reloaded by the second:
for i:=lo to hi do A(i); -> for i:=lo to hi do begin A(i); B(i) end;
for i:=lo to hi do B(i);
This is exactly neural-api's consecutive element-wise passes over one
TNNetVolume -- a bias/scale add immediately followed by an activation, or a
gradient accumulate followed by a weight update -- each otherwise a separate
memory-bandwidth-bound loop. The fused element-wise loop is left as a single
for-node so the vectorizer (which runs right after) can still pack it.
DEPENDENCE SAFETY (a wrong fusion is a miscompile; the recognizer is strict and
anything unmatched compiles exactly as before). Fusion is legal iff no iteration
i of loop 2 needs a value loop 1 has not yet produced for that same index i.
This is guaranteed by one blunt, provably sufficient rule instead of a general
dependence test:
* EVERY array-element reference in both bodies, read or write, is indexed by
*exactly* the loop counter (unit stride, no a[i-1]/a[i+1] offset). Then
everything either body touches at "time i" lives at array index i, so after
the reorder body1(i) still runs before body2(i) and no iteration ever
reaches an element another iteration owns. This holds regardless of
aliasing: two dynamic arrays can only fully alias at offset 0 or be disjoint
(a shifted overlap is impossible -- the same fact LOOPDISTPAT used), and a
full alias with identical [i] indexing is still element-wise safe; static
arrays / fields likewise, since the index is the same i on both sides.
* The ONLY writes allowed are to such an a[counter] element (the write flag
sits on the vecn; the check is on the assignment *target* shape, not a
write/modify flag, because writing a[i] marks the array base -- e.g. a
field-backed dynamic array Self.FData[i]:=x -- nf_modify too, and that base
is a safe element access, not a scalar write). A scalar / field / pointer
write is the channel by which a value could cross the loop boundary, so it
is declined. Because no scalar or field is ever written by either body,
every scalar/field a body reads (and every variable the shared bounds
mention) is loop-invariant across the whole fused region.
* No calls, no pointer derefs, no nested loops, no break/continue/goto/label/
exit/raise, only plain (:=) assignments; -Cr/-Co checked code is declined
(fusion reorders the per-element checks). Inline intrinsics are declined
except a whitelist of pure single-arg arithmetic ones (abs/sqr/sqrt and the
min/max family) -- Min/Max is what FPC's if-conversion turns a
`if a[i]<0 then a[i]:=0` ReLU activation into (in_max_*), the very
activation-after-scale shape this targets.
* Both loops ascending, unit step, bounds structurally equal (tnode.isequal)
and free of calls/derefs. Counters may be the same variable or two distinct
simple non-aliased ordinal locals; in the latter case loop 2's body keeps
its counter c2 and the fused body prepends c2:=c1 each iteration, which both
makes loop 2 see c2=i and leaves c2 holding hi if the loop ran / unchanged
otherwise, matching a stand-alone for-loop's post-value with no separate
guarded fixup (same-counter fusions carry no such assignment and vectorize
unchanged). The fused loop leaves loop 1's counter at hi as the original
first loop did.
* The pass greedily folds a whole run L1;L2;L3;... into one loop in a single
visit (the survivor keeps loop 1's counter; its already-checked,
counter-preserving body is trusted on the re-fold so the synthesized node's
missing DFA info is not re-queried). Non-loop statements between two loops
(only nothing-nodes are skipped) block adjacency.
New switch cs_opt_loopfuse (-OoLOOPFUSE), node pass OptimizeLoopFuse in
optloop.pas firing on statement-list nodes, call site in psub before the
vectorizer / strength reduction / the for->while lowering, added to
genericlevel4optimizerswitches so plain -O4 enables it. Per-loop diagnostics
06073/06074 (-vn) mirror the sibling loop passes. ppudump switch-name table
updated.
Benchmark (best-of, `array of single`, identical checksum on/off proving equal
output; -O4 fused vs -O4 -OoNOLOOPFUSE):
scale+bias then ReLU, 4 MB, 60 reps : 113 ms -> 97 ms ~1.16x
accumulate then weight-update, 3x16 MB, 25 reps : 266 ms -> 234 ms ~1.14x
Memory traffic drops by one array round-trip per fused pair: the intermediate
that loop 1 stored and loop 2 reloaded (a[i], resp. the gradient g[i]) now stays
in a register -- for accumulate-then-update, 6 array streams through DRAM per
rep fall to 5 (~17%), matching the measured speedup; the rest is bandwidth
bound.
Tests: unleashed/tests/testfiles/optfuse/ -- element-wise producer/consumer
(same and different counters, boundary lengths 0/1/2), post-loop counter value
(hi when run, unchanged when empty, for both counter shapes), a >2-loop greedy
chain and a same-then-different-counter mix, the field-backed TNNetVolume
bias+ReLU shape, the -O4 default set, and a negatives file asserting each
decline path (different bound, cross-iteration a[i-1], downto, non-whitelisted
Trunc intrinsic, scalar carry, non-adjacent statement, nested loop) still
computes the same result. Full unleashed suite 855 -> 861 pass / 6 fail (the 6
are the documented pre-existing failures); stage-2 -O4 self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the branch-predication -> vectorization pipeline for the element-wise
min/max activations neural-api runs over a TNNetVolume. FPC's existing -O2 if-
conversion (nflw.pas internalsimplify) already rewrites a branch-predicated
for i:=0 to high(a) do if a[i]<0 then a[i]:=0; { ReLU }
for i:=0 to high(a) do if a[i]>hi then a[i]:=hi; { one-sided clamp }
for i:=0 to high(a) do if a[i]<b[i] then a[i]:=b[i]; { element-wise max }
into a branch-free single-precision min/max intrinsic (a[i]:=max/min(...)), but
that scalar maxss/minss stayed a per-element operation the autovectorizer could
not widen (it only knew +,-,*). -OoIFCONVERT teaches the vectorizer's recognizer
the min/max activation shape and widens it across four SSE lanes to a packed
maxps/minps main loop plus a scalar tail, so the data-dependent per-element
branch disappears entirely and the loop streams at SIMD width. Under an AVX
fputype the VEX v-forms (vmaxps/vminps) are used.
CORRECTNESS (a wrong widening is a miscompile; the recognizer reuses the strict
OptimizeVectorize gate and anything unmatched compiles exactly as before):
* The body must be exactly one plain a[i] := max/min(u,v) assignment where the
destination is a single-precision dynamic-array element indexed by exactly
the loop counter, and the loop is an ascending unit-step counted loop over a
simple non-aliased signed 32/64-bit counter (all shared with the existing
autovectorizer checks; -Cr/-Co checked code and inline candidates declined).
* Only the single-precision intrinsics in_min_single/in_max_single are matched
(a double/integer min/max would change the per-lane precision). Each operand
u,v is independently either an array element of the counter (loaded as a
128-bit window) or a provably loop-invariant single scalar (broadcast once
into a [s,s,s,s] slot before the loop, as the arr_scalar shape already does).
* NaN / signed-zero fidelity: maxps/minps return their SECOND source operand
when a lane is unordered. The recognizer maps that second operand to opB --
the min/max node's parameter-list head, i.e. the same NaN-preferred value the
scalar maxss/minss uses -- and the packed body emits max/minps opB,opA so
every lane is bit-identical to the if-converted scalar result (verified for
NaN, +-0.0 and +-Inf on both the SSE and AVX paths). No reassociation, no
fast-math gate: this is a pure widening of an already-branch-free body.
New switch cs_opt_ifconvert (-OoIFCONVERT), added to genericlevel4optimizer-
switches so plain -O4 enables it; ppudump switch-name table updated. The pass is
driven by OptimizeVectorize (whose psub gate now fires on either -OoVECTORIZE or
-OoIFCONVERT): a new vok_minmax tvectoropnode kind (nbas.pas) carries the two
operand windows and an ismax flag, lowered by tx86vectoropnode.pass_generate_code
to MAXPS/MINPS (or the VMAXPS/VMINPS VEX forms). Per-loop diagnostic 06075/06076
(-vn) mirror the sibling loop passes.
Tests: unleashed/tests/testfiles/optifconvert/ -- ReLU / both one-sided clamps /
element-wise max checked bit-exact against a scalar recompute over every tail
residue 0..3 (SSE and AVX fputypes), NaN/signed-zero/Inf special-value fidelity,
the plain -O4 default set, and a negatives file asserting each decline path
(double precision, an open-array parameter, a two-statement two-sided clamp, a
downto loop, a shifted a[i+1] index) still computes the correct result. Full
unleashed suite 861 -> 866 pass / 6 fail (the 6 are the documented pre-existing
failures); stage-2 -O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…th gated)
Ports gcc's -freassoc / LLVM's reduction reassociation to FPC for the sum /
dot-product reduction shape that dominates neural-api's inner products and L2
norms over a TNNetVolume:
for i:=lo to hi do acc := acc + expr(i); (sum)
for i:=lo to hi do acc := acc + a[i]*b[i]; (dot product)
A single serial accumulator is a loop-carried dependency: iteration i+1's add
cannot start until iteration i's has retired, so the loop runs at one FP-add
latency per element however wide the machine is. The pass splits acc into K=4
INDEPENDENT partial accumulators, each summing every fourth element, combined
after the loop:
s1:=0; s2:=0; s3:=0; i:=lo;
while i <= hi-3 do begin
acc:=acc+expr(i); s1:=s1+expr(i+1); s2:=s2+expr(i+2); s3:=s3+expr(i+3);
i:=i+4;
end;
acc := (acc+s1) + (s2+s3); { combine }
while i <= hi do begin acc:=acc+expr(i); i:=i+1 end; { scalar tail }
The four chains are independent, so four adds are in flight at once and the loop
becomes throughput- rather than latency-bound (~3-4x on a long single/double
reduction on a 3-4-cycle-latency, 1/cycle-throughput FP adder), and for exactly-
representable data the result is bit-identical to the serial loop.
CORRECTNESS. Reassociating the additions changes the grouping and hence the FP
rounding, so a FLOATING-POINT accumulator is split ONLY under fast-math
(cs_opt_fastmath), matching gcc's -fassociative-math requirement; the switch is
in genericlevel4optimizerswitches but its FP action is gated on fast-math being
active, exactly as the task requires. INTEGER accumulators use two's-complement
addition, which is exactly associative even on overflow, so they are always split
(-Co checked code is declined since a wrap would trap on a different add). The
recognizer is strict; anything unmatched compiles exactly as before:
* ascending unit-step counted loop over a simple non-aliased signed 32/64-bit
counter (so i+1..i+3 and hi-3 cannot wrap for any index the loop reached);
* body is exactly ONE plain acc:=acc+expr / acc:=expr+acc whose target acc is
a simple non-aliased, non-address-taken, non-volatile local/value-param FP or
integer scalar (the single-statement shape guarantees acc and the counter are
written nowhere else in the loop);
* expr is side-effect free and never mentions acc: no call, assignment, address-
of, nested loop, control transfer, write/modify of any location, or non-pure
inline (only abs/sqr/sqrt and the min/max family are allowed) -- so
duplicating expr four times with the counter shifted i+1..i+3 reads the same
values the serial loop would and adds no side effect (aliasing is irrelevant:
nothing but the local acc is stored);
* -Cr/-Co checked code and provably tiny constant-trip loops (< 2*K) are left
alone; procedures with labels are skipped at the psub call site.
The counter-shift copies substitute each plain read of i with (i+delta) and
typecheck the new node immediately: the surrounding expr is a copy of already-
typechecked body nodes that do_firstpass will not re-descend, but (i+delta) has
the identical type as the plain read it replaces, so the ancestors' cached
resultdefs stay valid. New switch cs_opt_reassoc (-OoREASSOC), node pass
OptimizeReassoc in optloop.pas firing on for-nodes, call site in psub after loop
peeling and BEFORE strength reduction (which would rewrite the a[i] index nodes
into pointer walks the counter-shift could no longer find) and the for->while
lowering. Per-loop diagnostics 06077/06078 (-vn) mirror the sibling loop passes;
ppudump switch-name table updated.
Tests: unleashed/tests/testfiles/optreassoc/ -- FP dot/sum and integer sum split
results checked bit-exact against an address-taken (hence declined, strictly
serial) reference over trip counts 0..40 and 997..1000 (all hi-3 residues); the
fast-math gate (integer still split / FP left serial under -OoNOFASTMATH, both
correct); a fast-math tolerance test (the split single-precision dot product
within a few ULP of an extended sequential reference); and a negatives file
asserting each decline path (a non-inline call in expr, a downto loop, a two-
statement body) still computes the correct result. Full unleashed suite 866 ->
870 pass / 6 fail (the 6 are the documented pre-existing failures); stage-2
-O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -funroll-and-jam / LLVM's loop-unroll-and-jam to FPC for a perfect
(or near-perfect) two-level counted loop nest: the OUTER loop is unrolled by
factor K=4 and the four duplicated INNER loops are fused (jammed) into one inner
loop, so a value the inner body loads once (b[j] in a matmul-shaped nest
for i .. for j .. c[i]:=c[i]+a[i,j]*b[j]) is reused across the four unrolled
outer rows straight from a register instead of being reloaded every outer pass,
and a per-outer-iteration scalar accumulator is register-blocked:
for i:=lo to hi do --> i:=lo;
begin while i<=hi-3 do begin
s:=0; s0:=0; s1:=0; s2:=0; s3:=0;
for j:=lo2 to hi2 do for j:=lo2 to hi2 do begin
s:=s+a[i,j]*b[j]; s0:=s0+a[i ,j]*b[j]; { b[j] loaded }
c[i]:=s; s1:=s1+a[i+1,j]*b[j]; { b[j] reused }
end; s2:=s2+a[i+2,j]*b[j];
s3:=s3+a[i+3,j]*b[j]; end;
c[i]:=s0; c[i+1]:=s1; c[i+2]:=s2; c[i+3]:=s3;
i:=i+4;
end;
while i<=hi do begin { scalar remainder }
s:=0; for j:=lo2 to hi2 do s:=s+a[i,j]*b[j];
c[i]:=s; i:=i+1;
end;
This is exactly neural-api's register-blocked inner products / convolution nests
over a TNNetVolume (pletora/neural-api/neural/neuralvolume.pas). Unlike -OoREASSOC
the jam does NOT reassociate: each of the four accumulators sums exactly one outer
row in the original j order, so every produced value is BIT-IDENTICAL to the
untransformed nest for any input (no fast-math needed).
DEPENDENCE / LEGALITY (a wrong jam is a miscompile; the recognizer is strict and
anything not matched compiles exactly as before). Jamming the four inner loops is
the same reordering as fusing four copies of the inner body running on outer
indices i..i+3, legal iff those copies touch provably disjoint memory (no
loop-carried dependence across the outer loop). One blunt sufficient rule:
* The ONLY writes the outer body performs are (a) to a simple non-aliased LOCAL
SCALAR accumulator -- renamed to a fresh per-copy temp in copies 1..3 (the
register-blocking payoff) and required DEAD outside the whole nest (every
reference lies inside the outer for-node) so the rename cannot change any
observable value and a scalar that reduces ACROSS the outer loop is declined
-- or (b) to an ARRAY ELEMENT whose subscript chain mentions the outer counter
i, so copy k writes the i+k slice and the four stores are disjoint.
* The outer counter i may appear in the body ONLY as an exact array subscript
[i] (unit stride, no i+/-c offset, never in a scalar computation or the inner
bounds); enforced by counting that every read of i coincides with a bare [i]
subscript. Hence every i-touching access in copy k lives at i+k (disjoint
across copies), arrays not mentioning i are never written (read-only, shared
safely), and the inner iteration space is i-invariant so the jam is well-
defined.
* No calls, pointer derefs, address-of, non-pure inline intrinsics (only the
abs/sqr/sqrt and min/max whitelist), and EXACTLY ONE nested loop -- the inner
counted for, appearing as a direct statement of the outer body so the
prologue/epilogue split around it is unambiguous -- with no other loop; no
break/continue/goto/label/exit/raise/try; only plain (:=) assignments.
* -Cr/-Co checked code (global and per-region) is declined; both loops
ascending, unit step, over simple non-aliased signed 32/64-bit counters i<>j
(so i+1..i+3 and hi-3 cannot wrap for any index the loop reached). DFA proves
i unmodified in the body; procedures with labels are skipped at the psub call
site. Non-constant bounds are handled by snapshotting lo/hi once and emitting
the scalar-outer remainder; a provably tiny constant-trip outer loop (< K) is
left alone.
The counter-shift copies substitute each plain read of i with (i+delta) and each
accumulator read/write with its per-copy temp, typechecking every new node in the
substitution callback (the surrounding body is a copy of an already-typechecked
tree do_firstpass will not re-descend into, but each substituted node has the
identical type of what it replaced -- the REASSOC substitution gotcha).
New switch cs_opt_unrolljam (-OoUNROLLJAM), node pass OptimizeUnrollJam in
optloop.pas firing on for-nodes, call site in psub before the vectorizer /
strength reduction / the for->while lowering, added to
genericlevel4optimizerswitches so plain -O4 enables it. Per-loop diagnostics
06079/06080 (-vn) mirror the sibling loop passes; ppudump switch-name table
updated.
Tests: unleashed/tests/testfiles/optunrolljam/ -- a scalar-accumulator matmul row
and an array-element accumulator checked bit-exact against a strictly-serial
(descending-outer, hence declined) reference and an independent flat-loop absolute
reference across outer trip counts 0..8 (below factor, factor, every residue),
inner lengths 0..5, and larger sizes 200..205 hitting every main-loop residue; a
remainder file sweeping residues 0,1,2,3 with non-constant inner bounds; a
negatives file asserting each decline path (loop-carried outer dependence,
break in the inner body, offset a[i+1,j], address-taken accumulator, scalar
reduction across the outer loop) still computes the correct result; and an
-O4-default smoke test. Full unleashed suite unchanged at 861 pass / 19 fail --
the identical 19 (6 documented pre-existing plus runner-inaccuracy/environment
artifacts) fail on the pristine baseline with the same runner, so zero regression.
Stage-2 -O4 -Sew self-compile builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fpredictive-commoning to FPC. In a counted loop that re-reads
the same array location across successive iterations through a small window of
constant offsets -- the classic stencil a[i]:=b[i-1]+b[i]+b[i+1] and 1-D
convolution inner loops sliding a window over an array -- the value b[i+c]
loaded at iteration i is exactly b[i+c-1] as seen at iteration i+1. Instead of
re-loading every offset each iteration, the window is carried in a rotating set
of scalar temporaries and only the LEADING EDGE b[i+maxoff] is loaded per
iteration:
if lo<=hi then begin t0:=b[lo-1]; t1:=b[lo]; end; { guarded preheader }
for i:=lo to hi do begin
t2 := b[i+1]; { leading edge -- the only load }
a[i] := t0 + t1 + t2; { window reads served from temps }
t0:=t1; t1:=t2; { rotate down for the next iteration }
end;
This is cross-*sequential*-iteration reuse in a single loop, which LICM
(invariant loads only), same-iteration CSE and unroll-and-jam (cross-*outer*-
iteration reuse) each miss. Measured on the stencil above, per-iteration
memory reads of b drop from 3 to 1 (the two trailing offsets come from
registers). No reassociation: each produced value is BIT-IDENTICAL to the
re-loaded one (FP included -- values are only carried), so the transform is
exact for any input.
LEGALITY (a wrong commoning is a miscompile; the recognizer is strict and
anything not matched compiles exactly as before):
* The reuse base B is a plain 1-D dynamic or normal (non-packed) static array
VARIABLE (a direct load, so field/pointer-backed arrays are declined) with
an unmanaged 8/16/32/64-bit integer or Single/Double element, read as
B[i+c] for constant offsets |c|<=16.
* B must be READ-ONLY across the loop: declined if any store target resolves
(through the vec chain) to B's variable. Disjointness from the stores is
the fork's trusted array-identity disambiguation that LOOPFUSE / LOOPDISTPAT
/ UNROLLJAM already rely on -- a store to a different array variable A
(A<>B by symbol) is assumed not to alias B. An in-place stencil that stores
into B itself is thus declined (its store target resolves to B), sidestepping
the stale/fresh-value hazard entirely.
* The commoned offsets must form a CONTIGUOUS window [minoff..maxoff] (every
integer between appears as a B read) of width 2..8. Contiguity makes the
preheader safe: it loads B[lo+minoff .. lo+maxoff-1] and the loop's leading
edge loads B[i+maxoff]; every index either touches is one the ORIGINAL first
iteration i=lo (which reads the whole contiguous window) already read, and
B[i+maxoff] is read by the original iteration i, so NO load is introduced
that the original did not perform. The if lo<=hi guard suppresses the
preheader on an empty / too-short loop, so a zero-trip loop performs no
speculative or out-of-bounds load (a hard requirement under -Cr and at the
edges of an array). All body reads are unconditional (no if/case/short-
circuit and/or), so the whole window is genuinely read every iteration.
* The loop is ascending, unit step, over a simple non-aliased signed 32/64-bit
counter not modified in the body (DFA); -Cr/-Co, downto and non-unit steps
are declined. Bounds are snapshotted into temporaries evaluated once, and
the transformed loop stays a real for-node so the counter's post-value is
exactly what the original left.
* The body is built only from a whitelist of side-effect-free constructs --
plain (:=) assignments to an array element or a simple scalar, reads,
constants and pure arithmetic -- with NO call, pointer deref, address-of,
as-cast, nested loop, if/case, break/continue/goto/label/exit/raise/try,
short-circuit and/or, or non-pure inline intrinsic (only the abs/sqr/sqrt
and min/max whitelist). Procedures with labels are skipped at the psub call
site like the sibling loop passes.
The reload-to-temp substitution typechecks each new tempref immediately (the
surrounding body is a copy of an already-typechecked tree do_firstpass will not
re-descend into, and the tempref has the identical element type of the vecn it
replaces -- the REASSOC substitution gotcha).
New switch cs_opt_predcom (-OoPREDCOM), node pass OptimizePredCom in optloop.pas
firing on for-nodes, call site in psub before the vectorizer / strength
reduction / the for->while lowering, added to genericlevel4optimizerswitches so
plain -O4 enables it. Per-loop diagnostics 06081/06082 (-vn) mirror the sibling
loop passes; ppudump switch-name table updated.
Tests: unleashed/tests/testfiles/optpredcom/ -- a 3-wide double stencil, a
2-wide int32 sliding window and a 3-wide Single stencil checked element-wise
bit-exact against an independent reference across trip counts 0,1,2,3 (window
boundary and the guard) and sizes 200..205 hitting every residue; a negatives
file asserting each decline path (in-place store into the base, read+write same
base, call in body, conditional in body, downto, address-taken counter) still
computes the correct result; a -Cr/-Co file (pass declines, no new range error
at the window edges or trip count 0/1); and an -O4-default 5-wide stencil smoke
test. Full unleashed suite 874 -> 878 pass / 6 fail (the identical 6 pre-
existing failures on the pristine baseline, zero regression). The -O4 -Sew
stage-2 self-compile builds clean (the resulting -O4-self-compiled binary is
non-functional on pristine HEAD too, an unrelated pre-existing condition).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-sra to FPC. A local variable of record type whose address never escapes is split field-by-field into independent synthesized scalar temporaries (one tlocalvarsym per field, of the field's type), and every rec.field access is rewritten to a plain load of the matching temp. The fields then live in registers and feed the existing data-flow passes (constant propagation, DFA, dead-store elimination) instead of round-tripping through the stack frame -- the motivating pattern being small geometry/stride records and accumulator aggregates threaded through hot code. New switch -OoSRA (cs_opt_sra), on by default at -O4. Runs first in tcgprocinfo.TransformNodeTree, before constant propagation / DFA, so those passes see the scalar temps. Skipped for routines with inline assembler. The transform is gated by a conservative single-walk escape analysis (foreachnodestatic) per candidate record: a record is scalar-replaced only when EVERY load of it is the direct base of a field subscript. That one invariant, plus an address-of rule and a by-reference-argument rule, declines all the unsafe shapes: * address taken of the record or any field (@rec, @rec.field) * record or a field passed as var/out/constref * whole-record assignment (rec := other), function-result stores * whole record passed by value / as a method Self, typecasts of the record * FillChar/Default zeroing (passes @rec) and untyped move/fillchar * a `with rec do` that the front-end could not resolve to plain subscripts (a simple local-record `with` IS resolved and is safely handled) Structural gates decline the record type up front: unions, variant/case parts, `absolute`/overlapping-offset fields, packed/bitpacked records, and any record carrying a non-scalar or managed field (v1 handles flat records whose fields are all unmanaged register-sized ordinal/enum/float/pointer types -- ref-counting is never at risk because managed fields are declined). Assignment-target and read-modify-write markers (nf_write/nf_modify) are carried from the replaced subscript onto the fresh load so DFA still sees a definition there; the new load is typecheck+firstpassed in the substitution (the surrounding tree is already lowered, cf. the REASSOC nil-resultdef gotcha). Tests in unleashed/tests/testfiles/optsra/: sra_correct_01 (integer/float/pointer fields, read-modify-write, loop-threaded records -- results checked against an independent reference), sra_negatives_01 (every escape gate -- behavioural equality with the record left in memory), sra_constprop_01 (SRA enabling constant propagation and dead-store elimination down to a constant return). Full unleashed suite: identical PASS=878 FAIL=6 (the documented pre-existing failures) with and without the change; stage-2 -O4 -Sew self-compile builds clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fstore-merging to FPC. A straight-line run of adjacent narrow
constant stores to consecutive addresses off the SAME base register -- the
shape left behind by record field-by-field initialisation and small constant
fills that the code generator lowers to per-field "mov $const,off(base)" -- is
coalesced into one wider, naturally-aligned store whose immediate is the
little-endian composition of the component constants. Two adjacent byte stores
become one 16-bit store, two 16-bit stores one 32-bit store, two 32-bit stores
one 64-bit store, etc.
New switch -OoSTOREMERGE (cs_opt_storemerge), on by default at -O4.
Approach: an x86-64 assembler-list peephole (TX86AsmOptimizer.TryStoreMerge,
called from PostPeepholeOptMov). The peephole level is the pragmatic fit for
FPC here: at this point taicpu MOV records carry fully-resolved base+offset
reference operands and known constant sources, so the "provably consecutive
addresses off the same base" and "constants compose little-endian" tests are
direct, and no earlier tree pass sees the per-field stores as one object. It
also catches stores the front-end leaves unmerged (e.g. out-of-order field
initialisation) that a node-tree pass keyed on a single record variable would
not.
Legality (all gates are structural and local):
* The run is collected forward from p via GetNextInstruction (which skips only
reg-alloc / line-info markers). Collection stops at the first instruction
that is not itself a qualifying constant store to the same base register, so
a call, a label/jump, a memory read, a base-register overwrite, a volatile
ref or any other instruction between two stores terminates the run and those
stores are never merged together. Consequence: between the instructions
that actually get merged there is provably nothing but const-stores-to-base,
and such a store neither reads memory nor writes its own base register.
* A qualifying store is "mov $const, off(base)" with a plain base+offset ref:
real base register, index = NR_NO, no symbol/relsymbol, no segment override,
refaddr = addr_no, empty volatility set. Symbol-bearing refs (rip-relative
globals) are declined, so only register-based (stack / frame / pointer)
stores are touched.
* Any pair of stores in the run that overlaps at all aborts the whole attempt
(overlapping stores are order-sensitive); every surviving store therefore
writes a distinct byte range, so composing them and placing the one wide
store at p's slot preserves the final memory contents.
* A merged group must exactly tile an aligned window [w0, w0+tw), tw in
{2,4,8}, w0 the natural-alignment boundary at or below p's offset (floored,
handling negative %rbp-relative offsets), fully covered by >= 2 component
stores. Anchoring the window on p guarantees p is a component, so p is
widened in place (its offset lowered to w0 when p is the high half) and the
other components are removed.
x86-64 has no move of a full 64-bit immediate to memory, so a 64-bit merge is
emitted only when the composed value fits a sign-extended imm32; otherwise the
next smaller width is tried, so a general 64-bit constant group coalesces into
two 32-bit stores rather than four 16-bit ones -- still a win, same as gcc.
Unaligned stores are cheap on x86-64 but the natural-alignment window is kept as
a mild quality gate.
Tests in unleashed/tests/testfiles/optstoremerge/: correct_01 (out-of-order
byte inits coalesce to a 32-bit store whose imm bytes spell a tag asserted with
%CHECKBIN_HAS, plus a 64-bit merge; every byte read back individually),
disabled_01 (same source at -O4 -OoNOSTOREMERGE -- the plain code generator does
NOT merge this out-of-order shape, so %CHECKBIN_LACKS the tag; proves the
peephole, not codegen, is responsible), endian_01 (little-endian composition of
byte->word and word->dword merges verified byte by byte), negatives_01 (call
between stores, aliasing pointer write between stores, overlapping stores,
non-adjacent offsets -- all declined and behaviourally correct).
Full unleashed suite identical PASS/FAIL set with and without the change (the 6
documented pre-existing failures only); RTL rebuild clean; stage-2 -O4 -Sew
self-compile builds clean (the flow analyser's spurious "items may be
uninitialised" hint on the run buffer is locally silenced with {$warn 5036 off}
so -Sew stays clean).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port gcc/tree-switch-conversion.cc's jump-table / bit-test *clustering* to
FPC's case-node codegen. Where tcgcasenode previously picked ONE whole-
statement strategy (genjumptable / genlinearlist / genjmptree), the new
tcgcasenode.gencaseclusters partitions the sorted labels by dynamic
programming into an optimal mix of clusters and dispatches between them with
a balanced binary comparison tree:
- jump-table cluster: a dense run (>=4 labels, >=40% density, span capped)
lowered through the existing target genjumptable (holes -> else);
- bit-test cluster: several labels within an ALU-word span sharing at most
3 target blocks, lowered to one unsigned range check + "1 shl (x-low)" +
an AND-mask membership test per target (the classic
"case c of 'a','e','i','o','u'" shape becomes a shift+AND, not 5 compares);
- plain-compare singletons for the leftovers.
Wired into pass_generate_code ahead of the classic single-strategy
selection (gated on cs_opt_casecluster, only on the ordinal non-64-bit-alu
path where hregister/opsize/elselabel/jmp_lt/jmp_le/jumptable_no_range are
already set up); returns false and falls back to the tuned default lowering
when nothing beats one compare per label or only a lone jump table results.
The generic implementation uses only hlcg/generic helpers so it stays correct
on every target sharing this code path.
cs_opt_casecluster ('CASECLUSTER') added to globtype + ppudump and to
genericlevel4optimizerswitches, so plain -O4 enables it; -OoNOCASECLUSTER
disables. Constant-arm-to-static-array switch-conversion (gcc feature (c))
is left as a follow-up.
Robustness fixes over the initial draft: the DP span computation now bails on
TConstExprInt overflow (labels straddling the type range no longer wrap past
the width guards and mint an absurd table/mask); jump-table clusters run
genjumptable on a private copy of the selector register so its in-place
SUB does not clobber the value other dispatch-tree paths still read; the
per-cluster jumptable_no_range is saved/restored and only omits the bounds
check when the cluster covers the whole selector-type range.
Tests (unleashed/tests/testfiles/optcasecluster): exhaustive semantic
correctness across cluster boundaries, holes and type bounds at -O4;
a bit-test whose composed mask spells an ASCII tag asserted present via
%CHECKBIN_HAS and absent under -OoNOCASECLUSTER (%CHECKBIN_LACKS); signed
selectors with negative labels/ranges; -Cr/-Co range-checked shapes; and
negative cases (few labels, enum/boolean/string) that must stay correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fcrossjumping to FPC. When two or more predecessor blocks end
in an identical instruction sequence and converge on the same successor (the
if/else branches that share a trailing tail, case arms ending with the same
cleanup, and the per-arm Exit(TError...) boilerplate this repo's TResult style
produces in every codec), one copy of the shared tail is kept and the other
predecessors are redirected to jump into it. Purely a code-size / I-cache win.
New switch -OoCROSSJUMP (cs_opt_crossjump), on by default at -O4.
Approach: a late, whole-block assembler-list pass in the x86 optimiser
(TX86AsmOptimizer.DoCrossJump, run from an override of PostPeepHoleOpts after
the ordinary post-peephole). At this point taicpu records carry fully
resolved, register-allocated operands, so equality is operand-exact and the
transform has zero semantic risk: the merged instruction stream that executes
is byte-for-byte the one that executed before, only its storage is shared, so
calls, RIP-relative loads and everything else are handled with no special
casing precisely because they must match textually to be merged.
Algorithm:
* Collect the block's "anchors" -- unconditional jumps to a label and RET.
* Group anchors that are operand-exact equal (InstructionsEqualExact compares
opcode, size, condition, segment prefix and every operand; references via
RefsEqual, which also rejects any volatility). The earliest anchor of each
class is the canonical copy; each later identical anchor is a candidate.
* For a candidate, walk backwards from both anchors in lock-step
(GetLastInstruction) counting identical trailing instructions. The walk
stops at the first mismatch, at any referenced label (GetLastInstruction
returns those as a boundary, so a merged tail can never contain a side-entry
label), at any interior unconditional transfer, and via an overlap guard
that keeps the deleted region strictly after the kept region.
* If at least 3 trailing instructions match (a single shared instruction is
not worth a jump), a fresh label is inserted before the kept tail, a
"jmp <newlabel>" replaces the duplicated tail, and the duplicated tail is
removed with correct tasmlabel ref-counting (each removed jump decrefs its
target, the new jump increfs the new label).
Correctness guards beyond operand-exact equality:
* The duplicated region is refused (RegionSafeToDelete) if it physically
contains anything whose removal could corrupt debug or unwind information --
any label (a zero-jump-refcount label can still be referenced by DWARF line
tables, CFI/EH frame data or exception tables), any CFI directive, variable
location note, alignment, symbol or constant. Only instructions and
disposable bookkeeping (reg/temp-alloc, comments, line markers) may be
dropped. This is what makes epilogue tails carrying .cfi directives, and
tails bracketed by eh_frame .Lc labels, safe by declining to merge them.
* Merge artefacts (the inserted label and redirect jump) are themselves walk
boundaries, so repeated merges compose soundly.
Tests in unleashed/tests/testfiles/optcrossjump/: correct_01 (multi-exit shared
tails, every path validated against a reference at -O4), disabled_01 (same
source at -O4 -OoNOCROSSJUMP, equally correct), exceptions_01 (managed
ansistring tails inside try/finally with a raising arm -- ref counting and the
exception frame survive the merge), casecleanup_01 (case arms with a common
cleanup epilogue), negatives_01 (a one-operand difference and a live goto label
must not be merged across).
Full unleashed suite: 896 PASS / 6 FAIL, the 6 being the documented
pre-existing failures (3x composable_records_rtti_flatten needing the unbuilt
Rtti package, multi_var_init_double_01, inline_vars_inferred_array_string_
widestring_01) plus incfile_length_matches_size_02, which is a shell-runner
cwd artifact (it opens its own source by relative path; passes from its own
directory) and compiles at -O0 where the pass never runs. Code-size at -O4 on
the repo's own CLIs (only the app code is recompiled; RTL is prebuilt without
the pass): calc .text 360048 -> 360016 (-32 B), sed .text 339024 -> 338976
(-48 B); both binaries produce identical output with and without -OoNOCROSSJUMP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -freorder-blocks cold-region sinking to FPC. Lays out each
routine so the straight-line fall-through follows the likely (hot) edge and
sinks cold error/raise regions out of the hot path to the end of the routine.
A cold region is the code guarded by a conditional jump that jumps over it to
the hot successor and whose body unconditionally calls a runtime error/raise
helper (fpc_raiseexception, fpc_reraise, fpc_handleerror, assert, RunError,
range/overflow/divbyzero/invalidcast/object-check helpers). This is exactly
the shape this repo's TResult/raise/Assert style emits in every codec's guard
clauses. Sinking it makes the hot path branch-not-taken and packs the cold
raise boilerplate away from the I-cache-hot straight line.
New switch -OoBLOCKORDER (cs_opt_blockorder), on by default at -O4.
Approach: a late, whole-block assembler-list pass in the x86 optimiser
(TX86AsmOptimizer.DoBlockOrder, run from the PostPeepHoleOpts override right
after DoCrossJump). At this point taicpu operands are fully resolved and
register-allocated, so the relocation is a pure placement change: the same
instruction stream still executes, only its address changes.
Algorithm per routine:
* FindAnchor locates the last top-level unconditional control transfer (RET,
unconditional JMP, or a provably-noreturn helper call). Sunk regions are
spliced in immediately after it -- still inside the function's CFI/size
bracket, but unreachable by fall-through.
* For each conditional guard jump p to a label L, TrySinkAt walks the region
[p.Next .. L) requiring it to be straight-line (no interior branch/RET; a
CALL is allowed, it is how the raise fires), to actually contain an error
helper call, and to close exactly on L. Interior labels are permitted only
if alt_jump (plain jump targets that travel with the region and still reach
the one preserved exit); CFI/debug-range labels, alignment, symbols and
constants force a refusal so unwind/line tables are never disturbed.
* The region is relocated after a fresh out-of-line label, the guard is
inverted and retargeted at it (hot successor becomes the fall-through), and
a rejoin "jmp L" is appended only when the terminal helper can return
(assert); a provably-noreturn raise/error region falls off the end with no
rejoin. tasmlabel ref-counts are maintained (old target decref, new label
and any rejoin increfs).
* Bounded by MaxRegionLen / MaxTransforms and gated on CanDoJumpOpts so
pathological input can't blow up; rescans from the top after each transform
since the list and anchor change.
Tests in unleashed/tests/testfiles/optblockorder/: correct_01 (guard-clause
raises, a loop body holding a guarded arm, two independent cold arms in one
routine, a returning assert arm that keeps its rejoin jump, and nested
try/finally over a managed type with a raising arm -- every hot path validated
and every sunk arm still raises the caught exception), disabled_01 (same source
at -O4 -OoNOBLOCKORDER, byte-for-byte identical behaviour -- the control that
proves only placement changes), exceptions_01 (distinct exception classes sunk
from cold guards, matching handler and message must survive, finally runs
exactly once), negatives_01 (branchy routines with no cold region -- the pass
must find nothing to sink and change nothing).
Full unleashed suite: 900 PASS / 6 FAIL, the 6 being the documented
pre-existing failures (3x composable_records_rtti_flatten needing the unbuilt
Rtti package, multi_var_init_double_01, inline_vars_inferred_array_string_
widestring_01, and incfile_length_matches_size_02, a shell-runner cwd artifact
that compiles at -O0 where the pass never runs). Verified on parseDigit: with
the pass the guard becomes "ja .Lcold" falling through to the hot result and
ret, with the fpc_raiseexception region sunk past the ret; with -OoNOBLOCKORDER
the raise stays inline mid-routine. Both builds pass every test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-sink to FPC as the symmetric counterpart of the landed
-OoLOOPMOTION LICM: where LICM HOISTS a loop-invariant computation up out of a
loop, code sinking pushes a partially-dead computation DOWN to its single use.
A pure, side-effect-free assignment V := <expr> that immediately precedes an
if and whose value V is consumed on only ONE arm of that if (and is dead on the
fall-through after it) is relocated into that single arm, so every path that
never uses V stops paying for it (the size-check / early-guard preamble shape):
d := a*b + c; if guard then // then-arm never touches d
if guard then --> exit;
exit; d := a*b + c; // sunk: only reached when the
use(d); use(d); // guard falls through
New switch -OoSINK (cs_opt_sink), on by default at -O4. Node-tree pass
OptimizeCodeSink in optloop.pas over statement lists, call site in psub after
the DFA is (re)built -- it reads the if-successor's DFA life set to prove V dead
after the branch -- and before the vectorizer, mirroring the sibling loop
passes; skips procedures with labels so a goto cannot land between the
assignment and the arm it was sunk into.
LEGALITY (a wrong sink is a miscompile; the recognizer is strict and anything
not matched compiles exactly as before):
* V is a plain local/parameter of an unmanaged ordinal/enum/float/pointer type
whose ADDRESS IS NEVER TAKEN (no alias can observe the delayed store),
non-volatile, non-threadvar, same scope.
* The RHS is built only from constants, plain reads of such non-addr-taken
locals/params and non-trapping unchecked add/sub/mul/unary-minus/value-
preserving conversion -- no call, memory deref, indexing, division, checked
arithmetic or managed operator -- so it cannot raise, allocate, call or
write through an alias, and evaluating it later or not at all is invisible.
Because every operand is a non-addr-taken local/param, nothing the if
condition evaluates (even a call) can redefine an operand in between.
* The condition does not read V, exactly one arm reads V, the other does not,
and V is absent from the if-successor's life set -- so V's only remaining
consumer is the arm we sink into and no path that skips that arm needs V.
Adjacency (the assignment is the immediately preceding statement) makes the
"operands not redefined in between" test trivial: only the condition runs
between the two points and it cannot touch the operands. The relocation copies
the assignment into the chosen arm and leaves a no-op in its old slot, so the
statement-list links stay intact; DFA is redone afterwards.
Not handled (a missed opportunity is fine): sinking a value used only on the
fall-through PAST an early-exit guard (V stays live after the if there),
sinking across intervening statements, and sinking into case arms -- all left
for a follow-up; the current pass is the sound if/then-else subset.
ppudump switch-name table kept in sync. NB: this is the first optimizer switch
to grow sizeof(optimizerswitches) past the previous byte, which changes the
generic-token settings block serialised by tokenwritesettings/tokenreadsettings
in scanner.pas, so a stale prebuilt RTL trips "Wrong size of Settings read-in"
on generic-heavy units (fgl) until the RTL is rebuilt -- a normal consequence
of a tsettings size change that a full `make` handles.
Tests: unleashed/tests/testfiles/optsink/ -- a runtime-correctness fixture
covering then-arm, else-arm, self-reference (V:=V*V), both-arms (declined),
live-after (declined) and no-else shapes with a -OoNOSINK control proving
identical results; and assembly-placement fixtures (-al -s) asserting via
statement ordering that the sunk multiply is emitted AFTER the guard's branch
with the pass on, BEFORE it with -OoNOSINK, and stays before the branch for the
both-arms and operand-reassigned decline cases. Full unleashed suite 907 pass
/ 5 fail (the documented pre-existing failures: 3x composable_records_rtti on
the unbuilt Rtti package, inline_vars_inferred_array_string_widestring_01,
multi_var_init_double_01; zero regression). The -O4 -Sew stage-2 self-compile
builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -fgcse-sm / tree-ssa-loop-im "store motion" to FPC as the store-side
counterpart of the landed -OoLOOPMOTION LICM (which only HOISTS invariant
reads/pure expressions and never sinks a store). When a loop repeatedly
loads/stores a memory location whose ADDRESS is loop-invariant, the value is
promoted to a register temp for the loop's duration: it is loaded ONCE before the
loop, every body access is rewritten to the temp, and it is stored back ONCE
after the loop -- so the accumulation shape g:=g+<expr> stops re-loading and
re-storing g through memory on every iteration:
for i:=1 to n do t:=g; // one load before
g:=g+i*2 --> for i:=1 to n do
t:=t+i*2; // register-only body
g:=t; // one store after
New switch -OoSTOREMOTION (cs_opt_storemotion), on by default at -O4. Node-tree
pass OptimizeStoreMotion in optloop.pas over the still-structured for/while/repeat
nodes, call site in psub BEFORE ConvertForLoops lowers them (so the body is clean
user code, not compiler-generated temp/inline scaffolding) and, like the sibling
loop passes, skipped for procedures with labels so a goto cannot land between the
pre-load and the post-store.
SCOPE (v1): a plain module/unit GLOBAL (tstaticvarsym) of unmanaged scalar
(ordinal/enum/float/pointer) type that is WRITTEN at least once in the loop. A
global's address is a fixed static address, so pre-loading it before the loop can
never trap and never differs from the in-loop address, which sidesteps the
address-invariance/non-trapping-address proof that array elements a[k], pointer
derefs p^ and record fields would need -- those are left for a follow-up
(record-field promotion in while/repeat loops is already covered by
optimize_record_writes).
LEGALITY (a wrong promotion is a miscompile; the recognizer is strict and any
loop not matched compiles exactly as before). Because the promoted location is
no longer written inside the loop, nothing else in the loop may observe or
clobber it through a different path, and the post-loop store must reproduce
exactly what the in-loop stores would have left. The ENTIRE loop node
(bounds/condition AND body) is therefore required to be built only from a strict
whitelist (sm_node_safe) that provably cannot:
* call -- no calln/inlinen (a call may read/write the global);
* raise before the post-loop store would run -- no deref/index/vec, div/mod,
checked arithmetic or range-checked convert, no managed operator (a raise
would leave the global at its stale pre-loop value where the original left a
partial accumulation, observable in a handler);
* alias-write the global -- the only stores allowed are to plain
local/param/static simple lvalues (distinct non-addr-taken statics do not
alias; indexed/deref/absolute lvalues are rejected, declining the loop);
* alias-read the global -- all memory reads except plain scalar
local/param/static vars and non-aliasing compiler temps are rejected;
* transfer control out mid-loop -- procedures with labels are skipped, and the
whitelist admits no break/continue/exit/goto/raise/try nodes.
The candidate global is additionally required non-addr-taken, non-volatile,
non-threadvar, non-external and regable. (different_scope is NOT excluded: for a
static it merely records that the global is read from a subroutine rather than
its declaring scope -- always true for a global used in a procedure -- and adds
no aliasing path the no-call/no-deref whitelist has not already closed.)
ZERO-TRIP / EARLY PATHS: the temp is initialised to the global's current value
before the loop, so a zero-trip loop's post-loop store writes the identical value
back -- observationally a no-op for a non-volatile, non-aliased plain variable
(the store lands at the loop-exit label, reached both by the normal exit and by
the zero-trip skip). Conditional in-loop writes are safe for the same reason:
the temp mirrors the global exactly on every path.
ppudump switch-name table and the 'STOREMOTION' string table kept in sync. This
is the 50th toptimizerswitch; the 49th (SINK) already grew sizeof(optimizerswitches)
into its 7th byte, so this one does not change the serialised tsettings size and
needs no RTL rebuild.
Tests: unleashed/tests/testfiles/optstoremotion/ --
* optstoremotion_correct_01 (-O4): 23 checks over for/while/repeat
accumulation, zero-trip and negative-bound (global unchanged), single-trip,
conditional write, two globals promoted in one loop, a global loop bound,
double accumulation, and the decline shapes (call, aliasing pointer store,
break, indexed store, nested loop) -- each asserted against a hand-computed
compile-time constant so the oracle is never itself loop-optimised;
* optstoremotion_disabled_01 (-O4 -OoNOSTOREMOTION): the identical logic as a
control, proving results are byte-for-byte identical with the pass off;
* optstoremotion_placement_01 / _disabled_01 (-al -s, %CHECKASM_ORDER): the
global's load appears before the loop and its store AFTER the loop back-edge
with the pass on, and the store sits INSIDE the loop with it off.
Full unleashed suite 910 pass / 6 fail (the documented pre-existing failures:
3x composable_records_rtti on the unbuilt Rtti package,
inline_vars_inferred_array_string_widestring_01, multi_var_init_double_01,
incfile_length_matches_size_02 runner-cwd artifact; zero regression).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports gcc's -ftree-vrp / early-VRP to FPC as the branch-FOLDING consumer of value
ranges, complementing the two landed range passes: where -OoRANGEELIM spends its
ranges only on removing -Cr range-CHECK nodes and -OoJUMPTHREAD seeds facts only
from dominating branch conditions, VRP forward-propagates integer value INTERVALS
seeded from a variable's declared subrange/ordinal TYPE bounds and from for-loop
counter constant bounds (plus straight-line const/mod/and facts), then folds every
user-level if whose comparison the intervals already decide, deleting the dead
arm. The canonical win no other pass makes:
for i := 0 to 5 do for i := 0 to 5 do
if i > 10 then --> <else-arm> // i in [0..5] => i>10 false
<dead> // the whole then-arm gone
else <else-arm>
New switch -OoVRP (cs_opt_vrp), on by default at -O4 (globtype.pas + ppudump).
Pass OptimizeVRP in optloop.pas; call site in psub.pas placed FIRST among the loop
passes, before loop-splitting/peeling/lowering rewrite the still-structured
for-nodes' constant bounds into temps (so the counter interval can be read
directly), and refreshes DFA afterwards since it removes branches. Skips
procedures with labels like the sibling loop passes.
WHAT FOLDS (each recorded interval [lo..hi] for V is kept a *superset* of V's true
reachable value-set, which makes folding a comparison against it sound in both
directions):
* for-loop counter with constant bounds: V in [min(a,b)..max(a,b)] throughout
the body, when V is a simple non-aliased local/value-param not written there;
* a variable's declared ordinal/subrange TYPE range (get_min/max_value), applied
as an ever-present implicit fact -- e.g. var s:0..100; folds if s>150 ;
* straight-line seeds flowed within a statement list: V:=<const> gives [c..c],
V:=<a> mod <k> gives [-(k-1)..k-1] (k>0 const; sound even under truncation to a
narrower V), V:=<a> and <k> gives [0..k] (k>=0 const);
* the interval a dominating if V<op>c implies in each of its arms.
The declared type range and every flowed fact on the compared variable are
intersected into the tightest known interval and the comparison decided once
(vrp_interval_decides); a proven-true guard keeps only the then-arm, a proven-false
guard only the else-arm.
WHAT DECLINES (a wrong fold is a miscompile; the recognizer is strict and anything
not matched compiles exactly as before):
* only ordinal-integer comparisons V <op> const (=,<>,<,<=,>,>=) with V a
simple non-aliased, non-address-taken, non-volatile, non-threadvar local or
value-parameter (jt_recognize_cmp/rangeelim_simple_var, reused from the
range/jumpthread infrastructure) -- never floats, never a var an alias, call
or other thread could move outside its interval;
* the folded-away condition is itself side-effect-free (a plain compare of a var
against a constant), so deleting the dead arm removes nothing observable;
* a comparison never traps and the mod/and seeds only RECORD a range -- the
mod/and expression still executes -- so -Cq/overflow trap behaviour is
unchanged;
* intervals are invalidated conservatively: whenever a variable is (or may be)
written -- by an assignment, or anywhere inside a nested loop/if-arm/unmodelled
construct -- its interval is dropped from the fact set flowing past that write
(jt_writes_var); a for-loop counter interval is asserted only when the counter
is provably unwritten in the body;
* a V:=Length(a) => V>=0 seed is deliberately NOT taken: for a signed
narrower-than-SizeInt V a huge length truncates negative (unsound), and for an
unsigned V it is already covered by the type-range fold (redundant);
* genuinely overlapping intervals do not fold -- both arms survive.
v1 folds if-statements; case-branch elimination and while/repeat sequential
seeding are left as follow-ups (nested ifs inside case/try are simply not folded).
Reuses the range/jumpthread helpers (rangeelim_simple_var, rangeelim_const_value,
jt_recognize_cmp, jt_negate_relop, jt_writes_var) so the soundness gate matches the
sibling passes exactly.
Tests unleashed/tests/testfiles/optvrpfold/ (distinct from RANGEELIM's optvrp/):
for-loop false-arm deletion and true-guard else-deletion (%CHECKBIN), mod/and
derived folds with a -OoNOVRP control proving the fold is VRP's, a genuinely
overlapping range that keeps BOTH arms, a subrange-type fold, and a broad
Halt-oracle runtime test proving identical observable behaviour. Full unleashed
suite 917 pass / 6 documented pre-existing fails, zero regression. Adding this
54th optimizer switch left sizeof(optimizerswitches) unchanged (still 7 bytes), so
no settings-block resize and no RTL rebuild was needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Managed-type reference-count traffic elision (ARC-style pair elimination,
LLVM ObjCARCOpts / Swift retain-release / Delphi const-string idiom). A new
pre-firstpass node pass OptimizeRefElide (compiler/optloop.pas, switch
cs_opt_refelide in globtype.pas + ppudump) recognises a straight-line
ansistring borrow
a := b; { a a local, b a value-parameter or single-assignment local }
where the destination a is thereafter only READ -- never reassigned, never
address-taken, never passed by var/out -- and lowers it to a plain pointer
copy (PPtrUInt(@A)^ := PPtrUInt(@b)^, no incref) while marking a's localvarsym
(new transient field refelide_noinitfinal) so ngenutil skips its finalization
(no decref). Both the incref the assignment would perform and the decref the
implicit finally would perform are removed.
Soundness: the source b keeps the buffer alive for a's entire lifetime -- a
value parameter through the entry incref FPC does (held until scope-end para
finalization, so no aliasing store or exception can drop it early), a local
through its own reference (b written at most once => its buffer never changes,
held until b's own finalization). a therefore never owns a reference, so
eliding both operations -- including on exception-unwind paths -- is balanced.
The analysis is flow-insensitive (whole-routine nf_write/nf_address_taken
occurrence counts + the addr_taken/different_scope symbol flags), correct under
any control flow. const/var/out parameter sources are declined (a const source
is only borrowed from the caller and an aliasing store could free it mid-call);
chained borrows are declined; first cut is ansistring only.
Runs before do_firstpass lowers a := b into an fpc_ansistr_assign call
(psub.pas), gated on cs_opt_refelide, skipped for assembler routines. Landed
OPT-IN (NOT in genericlevel4optimizerswitches) given the severity of a wrong
refcount elision. Adding the 55th switch left sizeof(optimizerswitches)
unchanged (7 bytes), so no settings-block resize / RTL rebuild.
Tests unleashed/tests/testfiles/optrefelide/: runtime correctness with
dynamically-built heap strings (borrow from value-param and from local, the
alias-clear hazard, and the reassign/UniqueString decline shapes) verified
identical with the switch off (optrefelide_disabled_01), and a raise/except
path fixture. Manually confirmed via -al -s that the destination's incref and
decref are gone (source's pair untouched) and via -gh that allocated == freed
with zero unfreed blocks on every fixture. Full unleashed suite 920 pass / 6
documented pre-existing fails with -OoREFELIDE forced on all 923 tests, zero
regression.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…accumulated expression reassoc_subst_cb splices (i+delta) into expression copies that were already firstpassed as part of the original loop body, so every ancestor of the substituted counter read kept a cached resultdef and tnf_pass1_done keyed to the OLD child: a power-of-two multiply silently dropped the +delta (r:=r+i*2 summed wrong from trip count 4 up) and a non-power-of-two multiply died with internalerror 200306031. Reset resultdef + pass1/error flags on the whole substituted expression so do_firstpass rebuilds it consistently. New regression test optreassoc/reassoc_counterexpr_01 covers i*2, i*3, 2*i and (i+100) across trip counts 0..40; full unleashed suite green modulo the 5 known pre-existing failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the static-table half of gcc/tree-switch-conversion.cc (the switch_conversion
class behind -ftree-switch-conversion) to FPC, complementing the already-landed
-OoCASECLUSTER which only optimises the DISPATCH. When every arm of a
fully-covered case over an ordinal selector merely assigns compile-time constants
to the SAME ordered set of simple ordinal variables, the whole statement --
dispatch AND bodies -- is replaced by, per assigned variable, a static const
array indexed by (selector-low) plus a single range guard jumping to the else
part, eliminating all branching for the classic "map enum -> weight/flag" shape
(which a jump table still pays an indirect branch for):
case c of w := switchtbl[ord(c)]; { full coverage: }
cRed: w := 10; --> { no guard at all }
cGreen: w := 20; ...
New pass OptimizeSwitchTable in optloop.pas, wired into TransformNodeTree in
psub.pas next to the sibling loop passes (gated on cs_opt_switchtable, refreshes
DFA afterwards since it removes branches). It runs after do_firstpass, walks the
node tree for casenodes and rewrites each qualifying one in place, building fresh
untypechecked replacement nodes and letting a local firstpass lower them. The
per-variable lookup tables are emitted as ordinary local typed-const staticvarsyms
(ctai_typedconstbuilder into al_typedconsts), indexed with a plain vecnode.
SOUNDNESS (a wrong conversion is a miscompile; the recognizer is strict and
anything not matched compiles exactly as before):
* ltOrdinal labels, ordinal selector, >=2 arms;
* every arm, flattened, is nothing but plain V := <ordinal-const> stores to
simple writable non-external/non-typed-const/non-funcret local, static or
value/var-parameter ordinal variables;
* every arm assigns exactly the SAME ordered sequence of targets, so replacing
the arms with one fixed store order is correct even if two targets alias;
* the labels fully and contiguously cover [low..high] with NO holes, so the
single range guard exactly partitions in-range (table) from out-of-range
(else) -- a hole would wrongly index the table instead of taking the else;
* the covered span is small enough to afford the tables (<=4096);
* the selector is evaluated once into a temp BEFORE any store (matching the
original evaluate-selector-then-dispatch-then-store order), so a selector
with side effects, or a selector variable that is itself a store target,
stays correct. The index conversion runs only inside the guarded arm, so
an out-of-range selector value is never converted/indexed.
When the selector type is fully covered the else part is provably unreachable
and is deleted along with the dispatch; otherwise it is reused verbatim as the
range guard's else. Case arms cannot be entered by a goto, so the pass is safe
in the presence of labels.
cs_opt_switchtable ('SWITCHTABLE') added to globtype (enum + OptimizerSwitchStr +
genericlevel4optimizerswitches) and mirrored in ppudump; plain -O4 enables it,
-OoNOSWITCHTABLE disables. Two notes cg_n_case_switchtabled /
cg_n_case_not_switchtabled (errore.msg 06083/06084) report a firing and, past the
all-constant gate, a "not converted: <reason>" (too-large/sparse range, or holes)
at -vn. Adding this 53rd optimizer switch leaves sizeof(optimizerswitches)
unchanged (still 7 bytes), so no settings-block resize and no RTL rebuild.
First cut is deliberately conservative per correctness-over-coverage: ordinal
(orddef/enumdef) targets and hole-free contiguous coverage only. String/float
targets and hole handling via a "present" side-table are left as follow-ups.
Tests (unleashed/tests/testfiles/optswitchtable): a broad Halt-oracle proving
identical observable behaviour of an enum map (full coverage) and an integer map
with else across the full input range incl out-of-range; a multi-variable
enum->(weight,weekend) map lowered to two parallel tables plus a signed selector
with negative labels; a %CHECKBIN_LACKS proof that the dead else (unique marker)
is deleted when converted, paired with a -OoNOSWITCHTABLE %CHECKBIN_HAS control
proving the deletion is SWITCHTABLE's doing; and a negative test that a
non-constant arm and a holey label range both bail and stay correct. Full
unleashed suite: only the 5 documented pre-existing failures, zero regression.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the redundant sign/zero-extension elimination pass of gcc (ree.cc, the pass
behind -free that is default-on at -O2 there) to the x86-64 assembler backend as
a new post-peephole pass, gated on cs_opt_ree and enabled by default at -O4. The
existing Movzx* peepholes in aoptx86 only match adjacent instructions inside a
single basic block; byte/word-at-a-time loop bodies (parsers, base-N codecs) and
mask-then-widen idioms re-extend the same register with a definition several
instructions back that those peepholes never reach.
New DoRedundantExtElim walks the post-peephole asm list and deletes a movzx/movsx
whose value is already carrying the required extension. ExtAlreadyGuaranteed
does the analysis: for a reg->reg movzx/movsx into the SAME super-register (so
deletion is a pure no-op that cannot drop a distinct destination), it walks back
over a straight-line region with GetLastInstruction to the NEAREST definition of
the register and checks that this definition alone guarantees the extension
across at least the destination width. Qualifying definitions:
* a movzx/movsx of the same kind whose source width <= this one's and whose
destination width covers ours (a 32-bit-destination zero-extension also
clears bits 32..63 implicitly, so it covers a 64-bit zext target; a 32-bit
movsx does NOT sign-fill 32..63, so that widening is deliberately not applied
on the sign path);
* andl/andq $mask with a mask that clears every bit at or above the source
width (the mask-then-widen idiom, e.g. `andl $255,%eax; ...; movzbl %al,%eax`);
* xorl/xorq reg,reg (whole register zeroed);
* movl/movq $imm with a non-negative immediate below 2^width.
SOUNDNESS (a wrong deletion is a miscompile; the recognizer is strict and keeps
the extension for anything it does not model):
* only same-super-register extensions are deleted, so no value is lost;
* GetLastInstruction stops at referenced labels and block boundaries, i.e.
exactly where predecessors we cannot enumerate join, so a definition reached
only across a label makes the pass bail (no cross-block reasoning is claimed);
* calls are treated as clobbering definitions (RegModifiedByInstruction), so a
call between definition and use bails;
* the NEAREST writer must itself guarantee the extension: a partial write such
as `movb %al` leaves the high bits to an earlier definition we do not inspect
and therefore does not qualify;
* a 32-bit ALU result (add/mul/...) is not a qualifying definition for a byte
or word movzx, since it leaves bits 8..31 / 16..31 arbitrary; masks must be
non-negative and strictly below 2^width; movsx only accepts a movsx def.
cs_opt_ree ('REE') added to globtype (enum + OptimizerSwitchStr +
genericlevel4optimizerswitches) and mirrored in ppudump; plain -O4 enables it,
-OoNOREE disables. This 54th optimizer switch still fits in 7 bytes, so no
settings-block resize and no RTL rebuild.
Tests (unleashed/tests/testfiles/optree): a broad Halt-oracle proving identical
observable behaviour across the edge values that matter for extension width
(byte 0/127/128/255, shortint -128/-1/127) for a byte-sum loop, a shortint-sum
loop (movsx side), the mask-then-widen byte and word idioms, and a signed
round-trip; a conservativeness oracle where the extension is load-bearing (byte
extraction after a 32-bit add, a register serving two different extension widths,
and a definition separated from the use by a branch) that stays correct because
the pass keeps the extension; and a -OoNOREE control compiling the same redundant
shapes with the pass off, proving REE is behaviour-preserving. Manual -al check:
the two mask-then-widen movzx in the correctness test disappear with REE on (13
vs 15 movz) while the three load-bearing extensions in the conservativeness test
are all kept (3 vs 3). Full unleashed suite: only the 5 documented pre-existing
failures, zero regression; the new compiler bootstraps itself cleanly.
Cross-block reaching-definition reasoning (merging the extension into a
dominating definition in another basic block, and def-widening of narrow loads)
is left as a follow-up; the current cut is the conservative single-region
deletion subset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gue below a guard clause
Port gcc's shrink-wrapping pass (shrink-wrapping.cc, the pass behind -fshrink-wrap
that is default-on at -O2 there) to the x86 assembler backend as a new post-peephole
pass, gated on cs_opt_shrinkwrap. Instead of executing the callee-saved-register
saves unconditionally at function entry, the pass sinks them below an initial guard
clause so an early-exit fast path (nil check, zero-length check, cache hit, recursion
base) runs prologue-free and returns without ever saving -- or having to restore --
any callee-saved register.
New DoShrinkWrap recognises exactly one strictly-modelled shape and bails on anything
else (a wrong prologue move is a miscompile):
proc: proc:
push cs_1 .. push cs_k <guard region> ; volatile-only
<guard region> --> jcc Lfast ; fast exit
jcc Lexit push cs_1 .. push cs_k ; prologue now
...slow path... ...slow path... ; heads slow path
Lexit: Lexit:
pop cs_k .. pop cs_1 pop cs_k .. pop cs_1
ret ret
Lfast:
ret ; bare fast return
The prologue must be nothing but callee-saved integer pushes (no stack allocation,
no frame pointer) whose matching pops+ret is the epilogue jumped to. The guard
region is a maximal straight-line run of volatile-only instructions: no call, no
memory reference (so it cannot fault/raise or touch the stack), and it must not
modify RSP, the frame pointer or any register whose save is being moved -- which is
exactly what makes moving the pushes below it sound, since the saved incoming values
are unchanged. The guard's conditional branch is retargeted to a fresh bare ret
placed after the epilogue (replicating a callee-cleanup `ret $n' faithfully); the
pushes move to the head of the slow path.
SOUNDNESS:
* the fast path executes neither the pushes nor the pops and returns with every
callee-saved register still holding its caller value (it never touched one -- the
guard region is verified volatile-only), so the classic shrink-wrap miscompile
(a callee-saved register clobbered on the fast path) cannot occur;
* the slow path runs the byte-identical instruction stream;
* DWARF .debug_frame unwind data (always emitted on ELF via tf_needs_dwarf_cfi and
describing the exact prologue instruction offsets) stays correct: it lives in a
separate asmlist and only references the alt_dbgframe CFA position labels emitted
after each push, which are relocated together with the pushes, so the label-
relative advance_loc deltas of the FDE recompute to the correct new offsets.
Verified with objdump --dwarf=frames: the whole guard/fast-path range shows
CFA=rsp+8 with no registers saved, the saves appear only after the sunk pushes,
and the fast-path bare ret inherits the final fully-restored row;
* conservative bail-by-default whitelist: win64/nativent (SEH), inline CFI/SEH
directives in the proc code, exceptions / implicit finally frames, assembler
blocks, a stack-allocating or frame-pointer prologue, an interior referenced
(live) label, a memory operand or call in the guard, a foreign label, or an
epilogue that is not the matching pops+ret all make the pass decline;
* the guard length is capped so the first prologue CFA advance (often encoded as a
single-byte DWARF advance_loc1) cannot overflow after growing to span the guard.
Landed OPT-IN (-OoSHRINKWRAP), NOT in genericlevel4optimizerswitches: the transform
rewrites the uniquely sensitive prologue/epilogue, so per correctness-over-coverage
it is off at plain -O4 pending broader field exposure, even though it validated
cleanly (see below). cs_opt_shrinkwrap ('SHRINKWRAP') added to globtype (enum +
OptimizerSwitchStr) and mirrored in ppudump.
Tests (unleashed/tests/testfiles/optshrinkwrap, all with -OoSHRINKWRAP):
* _correct_01: guard-clause functions give identical results on the fast and slow
paths, recursion through a shrink-wrapped routine works, and a deep call chain of
them is correct, each checked against an independent oracle;
* _regsurvive_01: THE miscompile probe -- sentinel values the allocator must keep in
callee-saved registers survive 800 fast-path calls, and a mixed fast/slow loop is
checked against the identical arithmetic with the guard bodies inlined (no call);
* _bail_01: try/finally, raise/except, a nested procedure accessing the parent
frame, and an inline asm block all bail and stay correct;
* _disabled_01: the same shapes with -OoNOSHRINKWRAP produce identical results.
Manual -al check: reduce()'s guard testq/testl now precede the sunk pushq %rbx/%r12,
while the four disqualified functions keep their entry prologue. Full unleashed
suite: only the 5 documented pre-existing failures, zero regression, both opt-in and
with the pass force-enabled on every function; the compiler bootstraps itself cleanly
in both configurations (a full self-host with shrink-wrap active everywhere).
Follow-ups: multi-guard heads, frame-pointer / stack-allocating prologues (needs CFA
def_cfa_register / stack-adjust handling on the fast path), and promotion to
default-on at -O4 after field exposure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y elimination
Port the full-redundancy half of the gcc tree-fre / tree-pre (and LLVM GVN)
family to FPC as a new node-tree pass, complementing the existing intra-
expression CSE (-OoCSE / optcse.pas, which only commons repeated sub-trees of a
SINGLE expression) with redundancy ACROSS statements and across rejoining
branches -- and distinct from LICM (loop-invariant motion) and strength
reduction (index recurrences). When a side-effect-free scalar expression's
value is already available on every path reaching a point (computed on a
dominating statement / before a branch and reused on the rejoining arms, or
recomputed across straight-line unrolled bodies), it is computed once into a
temp and every later occurrence reads that temp instead of recomputing:
base := channel*size + 1; T := channel*size;
if cond then base := T + 1;
r := channel*size + base -> if cond then r := T + base
else else r := T - base;
r := channel*size - base; f := r + T;
f := r + channel*size; { one imul instead of four }
New pass OptimizeGVNPRE in optloop.pas, wired into TransformNodeTree in psub.pas
right before do_optcse -- the same late node-tree phase, after the loop
recognizers and ConvertForLoops, so it never disturbs their still-structured
for-nodes. Value numbering is structural (tnode.isequal); an available-
expression table is threaded through the straight-line statement flow, COPIED
into each arm of an if / case / loop and INTERSECTED at the rejoin, so a value
kept in the table is available on every path to a use, i.e. its definition
dominates the use. The temp is DECLARED unconditionally in a proc-entry
preamble and RELEASED at exit (balanced create/delete that dominate/post-
dominate every use, following the do_consttovar pattern), while the initialising
temp:=<expr> assignment stays at the first occurrence, whose operands are -- by
availability -- unchanged on the path to every use.
SOUNDNESS (a wrong reuse is a miscompile; anything not proven redundant compiles
exactly as before):
* only expressions over constants, non-address-taken / non-captured (no
vsa_addr_taken/vsa_different_scope) / non-volatile VALUE locals & params, and
memory reads (deref/vec/field), with a register-able scalar result, take
part; div/mod and managed/aggregate results are excluded;
* a local-only expression is killed by any assignment to one of its operand
locals (assignn target or inc/dec); a memory-reading expression additionally
by any store through memory or any call (both drop ALL memory entries), so
reads through a pointer are never reused across a store through any pointer;
* a value-expression that itself has side effects (nested call/store/asm/
side-effecting inline) is a barrier: no reuse inside it and the whole table
is cleared afterwards; an unmodelled mutating inline clears the table;
* the conditionally-evaluated RIGHT operand of a short-circuit and/or is never
GENERATED as available (only its unconditional left spine is), so p^ from
(p<>nil) and (p^>0) is not hoisted ahead of the nil guard;
* the caller skips procedures with labels, inline assembler or exceptions
(goto / partial-evaluation / regvar hazards).
cs_opt_gvnpre ('GVNPRE') added to globtype (enum + OptimizerSwitchStr) and
mirrored in ppudump. Opt-in (NOT in genericlevel4optimizerswitches) for the
first cut, per correctness-over-coverage and matching the recent opt-in siblings
REFELIDE / SHRINKWRAP: enable with -OoGVNPRE, typically alongside -O4. Note
cg_n_gvnpre_eliminated (errore.msg 06085) reports the firing count and the head
expression kind at -vn. PRE hoisting (making partially-redundant computations
fully redundant by sinking copies into predecessors), reuse within call
arguments, and by-reference/global-scalar participation are left as follow-ups.
Tests (unleashed/tests/testfiles/optgvnpre): a correctness oracle checking
cross-branch / straight-line / nested-if arithmetic reuse against an independent
reference over a full input range; a soundness oracle proving the traps are NOT
taken (intervening assignment, aliasing pointer store, call between global
reads, short-circuit nil-safety); a memory oracle for reused pointer deref and
record-field (FSizeX*FSizeY-style) products incl. correct reload after a field
store; and a -OoNOGVNPRE disabled control. Verified further by a 20k-case
randomized differential (-O- vs -O4 -OoNOGVNPRE vs -O4 -OoGVNPRE, all identical)
and by self-compiling the whole compiler with -OoGVNPRE active (816 firings on
its own source): that GVN-PRE-optimised compiler passes the full unleashed suite
with only the 5 documented pre-existing failures, zero regression.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…jects with -O4 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… vectorization
Widen the two canonical single-precision reductions
for i:=lo to hi do s := s + a[i]; (sum)
for i:=lo to hi do s := s + a[i]*b[i]; (dot product)
into a 4-lane packed SSE/AVX accumulator: a 16-byte slot is seeded with the
incoming scalar in lane 0 (xorps + movss) and zero elsewhere, accumulated
four elements per iteration (addps, or mulps+addps for the dot), horizontally
summed with a shufps+addps sequence (SSE2-only, no haddps) after the loop, and
the residue runs the original scalar body. On FMA-capable targets fast-math
contracts s+a[i]*b[i] into an fma() node before the vectorizer runs, so the
FMA-contracted dot shape is recognized too (widened to packed mulps+addps).
A partial-sum reduction reassociates the FP adds, so -- exactly like the
-OoREASSOC pass it mirrors -- it fires only under fast-math, and only when the
opt-in -OoVECTORIZE switch is set. OptimizeVectorize already runs before
OptimizeReassoc in psub and replaces the for-node with a block when it fires,
so REASSOC never re-splits a reduction the vectorizer took (the scalar tail is
a while-loop REASSOC ignores): the vectorizer wins with no pass-ordering
change. Soundness gates match the landed store shapes -- simple non-aliased
dynamic arrays of single, plain ascending unit-step counter, a simple
non-address-taken local single accumulator, single-statement body, -Cr/-Co
disable -- and double precision / two-statement bodies / address-taken
accumulators bail to correct scalar code.
The accumulator is memory-backed (the node-per-iteration body cannot hold a
register across the loop back-edge), so the speedup is compute-bound: ~1.25x
over plain scalar and on par with REASSOC on an L1-resident 512-element dot
product. A register accumulator, AVX-256 width and double precision remain
follow-ups.
New backend node kinds (tvectoropnode): vok_reduce_init / vok_reduce_sum /
vok_reduce_dot / vok_reduce_finish, with x86 codegen in nx86inl. New note
cg_n_loop_reduction_vectorized (06086). Tests in unleashed/tests/testfiles/
optvect/vect_reduce_* cover sum/dot for trip counts 0..40 plus large sizes
(oracle: a downto loop neither pass touches), nonzero incoming accumulator,
NaN/Inf propagation, the AVX2/FMA path, rejected shapes, the disabled control
and the -Cr disable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ment it in README Quick Start Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New -O4 tree-level optimization passes, x86 peepholes, and supporting fixes
I have not properly tested. This is for you to know what I am working on.
Summary
This PR adds a family of new optimization passes to the Free Pascal compiler, enabled at
-O4(each individually controllable via its own-Oo…/-OoNO…switch), together with x86-specific code-generation improvements, bug fixes uncovered by self-compiling the compiler at-O4, and an extensive regression-test suite underunleashed/tests/(216 files changed, ~28,700 insertions across 46 commits).New tree-level optimizer passes (
-O4)Loop optimizations (
compiler/optloop.pas, ~6,000 new lines)-OoLICM— loop-invariant code motion: hoists invariant computations out of loops.-OoLOOPUNSWITCH— loop unswitching: moves loop-invariant conditionals out of the loop, duplicating the loop body per branch (with a size cap).-OoLOOPPEEL— full peeling of small constant-trip-count loops.-OoLOOPSPLIT— loop splitting on an induction-variable crossover point.-OoLOOPFUSE— fusion of two adjacent loops iterating over the same space.-OoLOOPDISTPAT— loop-distribution pattern idiom recognition.-OoUNROLLJAM— outer-loop unroll-and-jam.-OoPREDCOM— predictive commoning of cross-iteration reloads.-OoIFCONVERT— loop if-conversion to packedmaxps/minps.-OoREASSOC— floating-point reduction reassociation (gated on fast-math).Vectorization
-OoVECTORIZE— conservative SSE loop autovectorization of element-wise array loops (add/mul/sub, copy, and scalar-broadcast shapes), with alias, range-check, and special-value safety guards. Includes per-loop diagnostics (notes 06065/06066) explaining why a loop was or was not vectorized.-OoVECTORIZEto the two canonical single-precision reductions,s := s + a[i](sum) ands := s + a[i]*b[i](dot product), widened to a 4-lane packed SSE/AVX accumulator with a scalar residue loop; recognizes the FMA-contracted dot shape on FMA targets. Fast-math gated (it reassociates FP adds, like-OoREASSOC). New note 06086.Value-range and idiom recognition
-OoRANGEELIM— value-range-analysis-driven range-check elimination (including dynamic-arrayHigh/Length-1bounds and must-check preservation for off-by-one and variable bounds).-OoVRP— interval value-range propagation with branch folding: forward-propagates integer intervals seeded from declared subrange/ordinal type bounds, for-loop counter bounds, and straight-lineconst/mod/andfacts, then folds everyifthe intervals already decide, deleting the dead arm.-OoBITIDIOM— recognition of bit-manipulation idioms: population count, andtzcnt/bsrbit-scan loops.modby a non-power-of-two constant: magic-number strength reduction (compiler/x86/nx86mat.pas).loopvar * invariantmultiplies inside array-index / pointer-offset expressions.Scalars, stores, and control flow
-OoSRA— scalar replacement of non-escaping local aggregates (new unitcompiler/optsra.pas).-OoSTOREMERGE— coalescing of adjacent narrow stores into wider stores.-OoJUMPTHREAD— jump threading / elimination of nested re-tests of the same condition.compiler/optdeadstore.pas) — DSE now also handles record-field and static-array-element stores.-OoSINK— partially-dead code sinking (the symmetric counterpart of LICM): a pure assignment immediately preceding anifwhose value is consumed on only one arm (and dead after the branch) is relocated into that arm, so paths that never use it stop paying for it.-OoSTOREMOTION— loop store motion / scalar promotion (gcc-fgcse-sm): a global of unmanaged scalar type repeatedly loaded/stored in a loop is promoted to a register temp — loaded once before the loop, stored once after — under a strict no-call/no-alias/no-trap whitelist over the entire loop.-OoGVNPRE(opt-in) — dominator-based global value numbering / full-redundancy elimination across statements and rejoining branches: a side-effect-free scalar expression available on every path to a point is computed once into a temp and reused. Availability tables are copied into branch arms and intersected at rejoins; killed conservatively by operand writes, memory stores, and calls. Validated by a 20k-case randomized differential and a self-compile with the pass active (816 firings). New note 06085.Case-statement lowering
-OoCASECLUSTER— gcc-style case clustering: partitions sorted case labels by dynamic programming into an optimal mix of jump-table clusters, bit-test clusters (thecase c of 'a','e','i','o','u'shape becomes one shift+AND membership test), and plain-compare singletons, dispatched via a balanced binary comparison tree.-OoSWITCHTABLE— case-to-lookup-table conversion (gcc-ftree-switch-conversion): a fully-covered, hole-free case whose arms only assign compile-time constants to the same ordered set of ordinal variables is replaced by per-variable static lookup tables plus a single range guard (no guard at all under full type coverage). New notes 06083/06084.x86 code generation
compiler/x86/aoptx86.pas): tail calls from a framed routine now reuse the caller's frame, with negative guards for managed types, pointer arguments, andtry/finallycontexts.-OoCROSSJUMP— cross-jumping / tail merging (gcc-fcrossjumping) as a late post-peephole pass: predecessor blocks ending in an operand-exact identical instruction tail keep one copy and jump into it. Pure code-size/I-cache win; refuses to delete regions containing labels, CFI directives, or debug/unwind data.-OoBLOCKORDER— static hot/cold basic-block layout (gcc-freorder-blocks): sinks cold error/raise regions (guard clauses ending infpc_raiseexception,RunError, range/overflow helpers, etc.) out of the hot path to the end of the routine, inverting the guard so the hot successor is the fall-through.-OoREE— redundant sign/zero-extension elimination (gcc-free) as a post-peephole pass: deletes amovzx/movsxwhose value already carries the required extension from its nearest definition (an earlier extension, anand-mask,xor reg,reg, or a small non-negative immediate move), reaching past the adjacency limit of the existing peepholes.-OoSHRINKWRAP(opt-in) — shrink-wrapping (gcc-fshrink-wrap): sinks a push-only callee-saved prologue below an initial volatile-only guard clause, so early-exit fast paths (nil check, zero-length, cache hit) run prologue-free with a bareret. DWARF CFI stays correct (position labels relocate with the pushes); validated withobjdump --dwarf=framesand a full self-host with the pass forced on.Managed types
-OoREFELIDE(opt-in) — managed refcount elision (ARC-style pair elimination): an ansistring borrowa := bwhereais thereafter only read is lowered to a plain pointer copy with the incref and the finalization decref both removed; the source provably keeps the buffer alive fora's lifetime. Verified leak-free with-ghon every fixture.Bug fixes and infrastructure
opttree: fixednormalize()hoisting block-expressions out of control-flow bodies, which broke-Oodeadstoreonif-expressions andmatchinside loops.REASSOC: fixed a miscompile / internalerror 200306031 when the loop counter appears in the accumulated expression — substituted(i+delta)copies kept stale cachedresultdef/pass1 flags; the whole substituted expression is now re-firstpassed.-O4stage-2 self-compile of the compiler build clean: fixed DFA false positives and case-exhaustiveness warnings.Doublefor real-literal initializers; documented theRttiunit build for tests.compiler/msg/errore.msg;ppudumpupdated for the new option flags.fpcu.sh(invoke the in-treeppcx64with the built unit paths) andlazbuildu.sh(lazbuildusing the in-tree compiler) wrapper scripts; README Quick Start section on compiling programs and Lazarus projects with-O4.Testing
unleashed/tests/testfiles/(optlicm,optunswitch,optvect,optvrp,optvrpfold,optbitidiom,optbitscan,optsibcall,optstoremerge,optdistpat,optreassoc,optcasecluster,optcrossjump,optblockorder,optsink,optstoremotion,optswitchtable,optree,optshrinkwrap,optgvnpre,optrefelide,dead_store_ext, and more), including must-fire, must-not-fire, safety, and disabled-switch cases.-O4with all new passes enabled (including full self-hosts with the opt-inSHRINKWRAPandGVNPREpasses forced on); the full unleashed suite passes with zero regressions beyond the documented pre-existing failures.Commits (oldest first)
02e9548dedadd -OoLICM loop-invariant code motion at -O4fa55bbe696tests: optlicm loop-invariant code motion coverage4b82685489add -OoLOOPUNSWITCH loop unswitching at -O43057f8705eadd -OoBITIDIOM bit-population-count idiom recognition at -O494c87769b4add -OoRANGEELIM value-range-analysis range-check elimination at -O4e2825fa5c9add -OoVECTORIZE conservative SSE loop autovectorizatione4345a9b90Signed mod by non-power-of-two constant: magic-number strength reductiona90bcb7ff0add tzcnt/bsr bit-scan idiom recognition to -OoBITIDIOM at -O46d845709f5strength-reduce loopvar*invariant multiplies inside array-index/pointer offsetsb848f80f2fadd -OoVECTORIZE per-loop vectorization diagnostic (notes 06065/06066)6f3d39b439-OoVECTORIZE: add scalar-broadcast and copy element-wise shapes5b5f4e3fc8optdeadstore: extend DSE to record-field and static-array-element storese996d0c5a9aoptx86: sibling-call frame reuse for tail calls from a framed routineb74d961688Add -OoJUMPTHREAD jump threading / nested re-test elimination at -O4b0c987dba5opttree: fix normalize() hoisting block-exprs out of control-flow bodies (-Oodeadstore)72c5194dadcompiler: make an -O4 stage-2 self-compile build clean (DFA false positives + case exhaustiveness)633d1110cbinline vars: infer Double for real-literal init; document Rtti unit build for tests (task 273)ff4de43e39add -OoLOOPDISTPAT loop-distribution pattern idiom recognition at -O46ba92cc480add -OoLOOPPEEL full loop peeling of small constant-trip loops at -O42593c865dcadd -OoLOOPSPLIT loop splitting on an induction-variable crossover at -O41c242376dbadd -OoLOOPFUSE loop fusion of two adjacent same-space loops at -O4c04aeae4a6add -OoIFCONVERT loop if-conversion to packed maxps/minps at -O4fc7657c34fadd -OoREASSOC floating-point reduction reassociation at -O4 (fast-math gated)3fd8c9f218add -OoUNROLLJAM outer-loop unroll-and-jam at -O43b6c0c915fadd -OoPREDCOM predictive commoning of cross-iteration reloads at -O41484d9878aadd -OoSRA scalar replacement of non-escaping local aggregates at -O4970e2eb2eaadd -OoSTOREMERGE adjacent narrow-store coalescing at -O46741dfc89cadd -OoCASECLUSTER gcc-style case clustering at -O46566e690f9add -OoCROSSJUMP cross-jumping / tail merging at -O4552e9aa493add -OoBLOCKORDER static hot/cold basic-block layout at -O482d2a479abadd -OoSINK partially-dead code sinking at -O47d17981cf2add -OoSTOREMOTION loop store motion / scalar promotion at -O4467c2c75e2add -OoVRP interval value-range propagation with branch folding at -O4f3b814abf0add -OoREFELIDE managed refcount elision (opt-in)e310cbeff3fix REASSOC miscompile/internalerror when the counter appears in the accumulated expressione6f1fcff78add -OoSWITCHTABLE case-to-lookup-table conversion at -O47eb098c537add -OoREE redundant sign/zero-extension elimination at -O48d2e851104add -OoSHRINKWRAP shrink-wrapping (opt-in), sinking a push-only prologue below a guard clause6884f2979aadd -OoGVNPRE dominator-based global value numbering / full-redundancy elimination9a7215bf4fadd Quick Start section to README: compiling programs and Lazarus projects with -O470e80d28e0add fpc.sh wrapper: invoke in-tree ppcx64 with -O4 and built unit paths3a9bda1b2bextend -OoVECTORIZE with single-precision sum / dot-product reduction vectorization