lexer: slice tokens from the source binary; drop per-char position maps

[davydog187]

Rewrites the hot paths of lib/lua/lexer.ex. Two commits: the first is a
behavioral fix to position tracking, the second is a pure-performance
rewrite on top of it.

Why

The lexer allocated far more than it scanned. Per source character it
rebuilt a three-key position map (advance_column/2, plus sixteen inline
%{line: ..., column: 1, byte_offset: ...} literals), and it grew every
token's text one byte at a time with acc <> <<c>>. On top of that, each
identifier was compared against the 22 reserved words in sequence and then
run through String.to_atom/1, and each number was classified with three
String.contains?/2 scans and a regex before parsing.

What changed

Position is threaded as three bare integers — line, the byte offset the
current line's columns are measured from, and the current byte offset — and
a position map is materialized only when a token or an error is emitted
(~140 maps instead of ~800 for a 370-byte input). The column base absorbs
the continuation bytes of multibyte codepoints, so column still counts
codepoints while byte_offset counts bytes; the "multibyte character as one
column" tests are unchanged.

Token text is binary_part/3 out of the source binary. String escapes
and long-string end-of-line normalization are the only cases where a token's
text differs from its source bytes, so those collect iodata chunks around
just the rewritten spans — an escape-free string, an identifier, a number,
and a comment are each a single slice. A bare \n inside a long string is
already the normal form, so it does not even break the span.

Keywords are pattern-matched function heads, which the compiler turns
into a binary decision tree; a non-keyword identifier now falls through with
no sequential comparison and no atom conversion.

Numbers carry their shape (integer / fraction / exponent) out of the
scanner and go straight to :erlang.binary_to_integer/1 and
:erlang.binary_to_float/1. The mantissa is normalized in the scanner,
where the shape is known, rather than by regex afterwards. The
>= 2^63 → float decimal rule and wrap_int64 for hex literals are
unchanged, as is the {:invalid_number, pos} error for 1e, 1e+, and
out-of-range magnitudes such as 1e400.

The bug (first commit)

Three scanners emitted a token and resumed without accounting for bytes they
had just consumed:

• a short string did not advance past its closing quote, so every later
  token on the line was one byte short per preceding string — local x = "ab" ]
  reported ] at column 15 instead of 16;
• a multi-line comment's body was assumed to start after --[ plus the
  level's = signs, forgetting the second [;
• a \u{...} escape charged 4 + digits columns, but digits already
  counts the closing brace, so it over-advanced by one.

The net effect was that {:eof, pos} did not report byte_size(source) and
anything slicing the source by byte_offset picked up shifted text. Line
numbers were never affected, so the luac line-number differential
convention is untouched. No existing test pinned the wrong values;
regression tests are added for all three.

Verification

• mix format clean; full suite green including --include slow --include differential --include lua53 (2642 passing).
• Behavior preservation. A scratch harness tokenized every .lua file
  in the repo (68 files: fixtures, the integration corpus, the Lua 5.3
  official suite) plus 24 hand-written edge cases — escapes, \z, mixed
  \r/\n/\r\n/\n\r, hex floats, malformed numbers, unterminated
  tokens, invalid UTF-8, shebangs — with the rewrite and with the previous
  lexer carrying only the three position fixes. Every token stream is
  byte-for-byte identical (68/68 files, 24/24 cases).
• Position correctness. An independent checker re-derives each token's
  line, column, and byte offset from the source and confirms identifiers and
  keywords slice back to their own text and that eof.byte_offset == byte_size(source). The rewrite passes 65/65 checkable files; the lexer on
  main fails 59 of them.

Numbers

Prod build, median of 20 batched runs, on the playground snippets and two
real files. Absolute times move a few percent between runs with machine
load; the ratios were stable across repeated runs.

input	bytes	before	after	speedup
metatables	370	29.10us	5.46us	5.3x
fib	180	13.77us	2.50us	5.5x
tables	256	21.02us	4.15us	5.1x
literals.lua	10,339	509.33us	84.53us	6.0x
assertions.lua	14,361	1189.4us	151.40us	7.9x

The metatables snippet's lexer stage goes from ~29us to ~5.5us, against a
target of 7us.

Scope is lexer.ex and its tests only; parser and compiler are untouched.

[davydog187]

🤖 Automated review — Opus + Sonnet consolidated findings

Two independent AI reviewers (Opus, Sonnet) reviewed this PR. The three claimed position fixes are real and correct (verified by hand-tracing and by a 200k-input differential fuzz), number/string/comment scanning has no semantic drift, line numbers are identical to main across all 76 .lua files in the repo, and the PR #395 valid-UTF-8 error invariant holds. Two substantive findings:

🔴 HIGH — single-line comment start column goes to zero/negative on multibyte input (regression)

lib/lua/lexer.ex:331-378 (scan_single_line_comment/8 → single_comment/4)

The comment's start position is materialized after the body is scanned, from the already-mutated line_start — so every multibyte codepoint inside the comment subtracts 1 from the reported start column:

• Lexer.tokenize("-- é\nx") → comment at column: 0 (main: 1)
• "-- ééé\n" → column: -2; "--é🎉é🎉\n" → column: -7

Confirmed by a 200,000-input randomized differential against this PR's own first commit: 2,400 mismatches, 100% of them this bug, 0 of any other kind. It violates the module's own @type position :: %{column: pos_integer()}, and the bogus column is publicly observable — lib/lua/parser/comments.ex:23,44 copy it into Lua.AST.Meta leading/trailing comments. The other three scanners are unaffected because they pass an eagerly-built start_pos; this is the only lazily-computed start position that can cross a multibyte codepoint. Fix the same way: thread the start position (or original line_start) eagerly.

🟠 MEDIUM/HIGH — token slices >64 bytes pin the entire source binary (both reviewers, independently measured)

lib/lua/lexer.ex:88-92, 281, 377, 386, 444, 698 (binary_part fast paths)

On this repo's OTP, binary_part/3 results over 64 bytes are sub-binaries: measured :binary.referenced_byte_size/1 of a 200-byte string literal sliced from a 164 KB source = 164,448 (on main: 200). Nothing downstream copies (git grep ':binary.copy' lib/ is empty; tokens flow into the AST → compiled chunk → runtime values), so one retained >64-byte literal/comment/identifier keeps the whole source resident for the life of the chunk — e.g. require-ing a 2 MB library pins 2 MB. The chunked path (strings with escapes) already copies via IO.iodata_to_binary; only the single-slice fast paths retain. Suggested fix: :binary.copy/1 on the slice fast paths — cheap at these sizes and keeps the throughput win.

🟡 LOW

• lib/lua/lexer.ex:490-491 — the value > 255 guard in the \ddd escape clause is dead code (read_decimal_escape/3 stops before next > 255). Related pre-existing divergence: PUC rejects \256 ("decimal escape too large"); we lex \25 + literal 6. Will be filed as an issue.
• Pre-existing (unchanged by this PR, confirmed identical on main, filed as an issue): bare \r inside a --[[ ]] comment doesn't advance the line counter (PUC does); bare \r inside a short string is accepted (PUC errors "unfinished string").

Test-coverage notes

• The HIGH is exactly the untested corner: the suite has multibyte-column tests for strings, long strings, and multi-line comments — but not single-line comments.
• A cheap column >= 1 property over the existing fixture corpus would catch this entire class.
• If the sub-binary behavior were kept deliberately, the moduledoc should say so — but a :binary.referenced_byte_size/1 assertion is the better pin.

Fixes for the HIGH, the retention issue, and the dead guard will be pushed to this branch shortly.

[davydog187]

Pushed 81c2988, which addresses the automated review's HIGH findings plus the retention issue. scan_single_line_comment/8 was building its start position after scanning the body, so multibyte codepoints in a comment skewed the reported start column ("-- é\n" gave column 0, "-- ééé\n" gave -2); it now builds the position eagerly like the other three scanners. The binary_part/3 single-slice paths (string and long-string bodies, identifiers, both comment kinds) now :binary.copy/1 their text, so a >64-byte token no longer keeps the entire source binary alive. The unreachable value > 255 branch in the \ddd escape clause is gone, with \256 lexing unchanged. Regression tests cover the comment columns, a line >= 1 and column >= 1 invariant over every token in all 29 test/lua53_tests/*.lua fixtures, and :binary.referenced_byte_size/1 == byte_size/1 for large string/comment/identifier tokens. Timing on the largest fixture (api.lua, 32KB), interleaved rounds so noise hits each variant equally: main 7283-7761 us min / 9640-18754 us median, branch pre-fix 545-772 us min / 1114-1721 us median, branch with these fixes 603-653 us min / 878-1460 us median — the copies are inside run-to-run noise and the hot-path win holds. mix test --include lua53 (2644 passed), mix format --check-formatted, mix compile --warnings-as-errors, and mix dialyzer are all clean.