Replace pest expression grammar with a Pratt parser and add first-class function support#13
Merged
Merged
Conversation
The pest-based expression parser ran out of stack on Windows. The recursive descent consumed one boxed `Pair<Rule>` per grammar level, and the grammar had grown deep enough (struct inits, array indexing, range literals) that 1MB of stack was no longer enough. Replace it with a hand-written Pratt parser that operates on a token stream instead, so recursion depth tracks expression nesting rather than grammar-tree depth. Range literal (a...b) support moves into the Pratt infix layer with binding power 2/2 (looser than assignment, tighter than logical or) and produces Expr::RangeLiteral, matching the pest grammar's range_expr structure. Summary of changes: - Re-add logos to dependencies - Introduce Tok, SpannedTok, tokenize, and tok_name in lexer.rs - Add kitlang/src/codegen/parser/diagnostics.rs - Add kitlang/src/codegen/parser/expr_pratt.rs, containing Pratt core - Wire Parser::parse_expr to PestExpr -> ExprParser pipeline - Add Tok::Ellipsis as a Pratt infix op building Expr::RangeLiteral - Move struct-init / field-init semantics into the Pratt path - Delete kitlang/src/codegen/parser/expr.rs (pest-based parse_expr) - Drop dead parse_bool_literal and unescape helpers from parser/mod.rs - Add range-literal unit tests: * range_literal_simple * range_literal_with_expressions - Mark diagnostic seams with #[allow(dead_code)] for the follow-up that will wire them in
The single 1200-line expr_pratt.rs mixed parser core, primary expression parsing, and tests, making it hard to navigate. The diagnostics module also duplicated its token-to-name mapping between tok_name and a local expected_name helper. Summary of changes: - Split expr_pratt.rs into mod.rs, primary.rs, and tests.rs via include! - Make diagnostics::expected_name the single canonical token-name match and derive tok_name from it instead of duplicating the match - Remove PlusPlus/MinusMinus from prefix binding power (not prefix ops) - Drop now-unneeded #[allow(dead_code)] attributes on used items
The lexer now properly reports unrecognized characters and integer overflow instead of silently dropping them. The parser validates no trailing tokens remain and provides better EOF error messages. Unsupported constructs like indirect calls are now explicitly rejected with clear diagnostics. Summary of changes: - Change tokenize() to return Result and report lexical errors - Add LexicalError enum for UnexpectedCharacter and IntegerOverflow - Validate trailing tokens in parse_kit_expr() - Distinguish UnexpectedEof from UnexpectedToken in errors - Reject indirect function calls with clear diagnostic message - Fix binding power checks in assignment operator handling - Update expect() to return UnexpectedEof when at EOF - Improve documentation accuracy and clarity - Add comprehensive test coverage for all error cases
kitc now supports passing functions as first-class values, enabling higher-order programming patterns. Functions can be passed as arguments, returned from functions, and stored in variables. The parser, type system, and code generator were updated to handle function types throughout the compilation pipeline. Summary of changes: - Change Call expression callee from String to Box<Expr> for indirect calls - Add Function type variant to support function-type values - Update Pratt parser to allow and parse indirect call expressions - Implement type inference for indirect calls via three paths: named, indirect, C interop - Register functions as first-class values in symbol table with function type - Add type unification logic for function types with parameter matching - Add function pointer parameter formatting with correct C declarator syntax - Support transpiling both named and indirect function calls - Remove parse-time rejection of indirect calls; validate at type-check time - Add test example demonstrating indirect function calls - Update parser tests to reflect indirect call support - Remove unused expr_to_callee_name function and replace with callee_name
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The expression grammar was implemented as 13 mutually recursive pest rules (
expr -> assign -> logical_or -> ... -> primary), one per precedence level. Because each level recurses into the next, deeply nested or long expressions could blow past the default 1 MB stack on Windows, causing the compiler to crash instead of reporting a parse error.Separately, the language had no way to treat a function as a value. Calls were parsed as a bare callee name (
Expr::Call { callee: String, .. }), so expressions like passing a function as an argument and invoking it indirectly (g(f, 4)) could not be represented, let alone type-checked or transpiled to C function pointers.Solution
Introduced a hand-written Pratt parser that replaces the old pest-based recursive-descent expression rules with a single binding-power loop. This reduces the call stack depth for chained/nested expressions.
Precedence and associativity are now defined declaratively, and parser-internal errors are represented as a small structured
ExprParseErrortype that is only converted into the publicCompilationErrorat one seam. Pest now hands eachexprpair's source text to the new parser.Added a Logos-based lexer to tokenize expression source text for the Pratt parser, with byte spans on each token to support future diagnostics.
Changed
Expr::Call.calleefrom a plainStringto a boxedExpr, so any expression can be called.A
callee_namehelper recovers the simple/qualified name for the common cases (named functions, etc.) while indirect calls fall back to inferring the callee's type.Added a
Type::Functionvariant & parsing of function type annotations, so functions can be registered as first-class values in the symbol table and passed around.Updated code generation to emit proper C function ptr syntax (
ret (*name)(params)) for function-typed parameters, variables, and struct/enum fields, and to transpile indirect calls as(callee_expr)(args).Added an
indirect_calltest trying a function passed as an argument and invoked through a parameter.