feat(codegen): add lex-only generated helpers - #316
Conversation
Generate lex and lex_stream conveniences in every lexer module so callers can inspect token text, type, and channel without constructing a parser or wiring a TokenSink. Return the eagerly buffered CommonTokenStream to preserve token ownership, source diagnostics, hidden and custom channels, and EOF. The generated source uses existing revision-6 runtime APIs, so the compatibility revision remains unchanged. Document the token-dump workflow, cover custom streams and channels, and refresh all checked-in recognizers.
Seed the nested build-dependency workspace from the repository Cargo.lock before running it offline. This keeps its dependency graph identical to the tested workspace and allows Cargo to reuse locked packages whose versions are subsequently yanked, including wide 1.6.0.
Copy/Paste DetectionFound 7 duplication(s) across 5 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 26 line (142 tokens) duplication in the following files:
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");
```rust
---
Found a 25 line (115 tokens) duplication in the following files:
* Starting at line 4057 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4267 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))Found a 22 line (112 tokens) duplication in the following files:
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
```rust
---
Found a 27 line (110 tokens) duplication in the following files:
* Starting at line 3411 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 3523 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
decision: 0,
alts: (1, 2),
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
plus_loop: false,
fast_path: None,
body: &body,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
insta::assert_snapshot!(Found a 25 line (104 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: false,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust
---
Found a 28 line (102 tokens) duplication in the following files:
* Starting at line 3204 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 3364 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// One decision renders into a fresh String; snapshot the whole emitted control flow (the
// semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
// instead of six positive probes plus one negative guard.
insta::assert_snapshot!(Found a 16 line (101 tokens) duplication in the following files:
atn.set_loop_back_state(3, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })
```rust |
|
Claude finished @tinovyatkin's task in 18m 50s —— View job Code Review — PR #316
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
📊 Source Code Metrics (this PR vs
|
| File | Cyclomatic | Cognitive | Functions | LLOC | MI |
|---|---|---|---|---|---|
| crates/antlr-rust-codegen/src/generator/tests.rs | 289 (main: 288) 🔴 | 48 ⚪ | 202 (main: 201) 🔴 | 1341 (main: 1337) 🔴 | 0 ⚪ |
| crates/antlr-rust-codegen/src/lexer/render.rs | 90 ⚪ | 77 ⚪ | 9 ⚪ | 145 (main: 144) 🔴 | 0 ⚪ |
| crates/antlr-rust-codegen/src/lexer/render_model.rs | 24 (main: 23) 🔴 | 9 ⚪ | 9 (main: 8) 🔴 | 35 (main: 34) 🔴 | 21.86 (main: 25.05) 🔴 |
Generated by mehen v1.8.1 — the code quality watcher.
Document vocabulary-resolved token names, always-visible numeric channels, buffered lexer diagnostics, and the CommonTokenStream construction panic. Move the README section outside the parser setup narrative. Exercise skipped rules and UTF-8 byte/column positions end to end, strengthen the legacy CommonToken guard, and let the lexer convenience snapshot own its rendered value. Regenerated recognizers retain generated-code API revision 6 while their stale generator banners now reflect the workspace's current 0.31.0 release.
|
Warning Review limit reached
Next review available in: 53 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughGenerated lexers now provide ChangesLexer convenience workflow
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant GeneratedLexer
participant CommonTokenStream
Caller->>GeneratedLexer: Call lex or lex_stream
GeneratedLexer->>CommonTokenStream: Construct token stream
CommonTokenStream->>GeneratedLexer: Request and buffer tokens
GeneratedLexer-->>CommonTokenStream: Return tokens and source diagnostics
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Addressed all six follow-ups in
The follow-up also records the generated-banner refresh in the durable commit message. Focused tests, clippy, strict rustdoc, regeneration checks, self-host fixed-point tests, Markdown lint, and generated hashes are clean. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/lexer.rs`:
- Around line 263-273: Update the generated-project test around the token
assertions to use Insta: add the pinned `insta = "=1.48.0"` entry as a generated
dev dependency and replace the manually specified observed token array with
`assert_debug_snapshot!`. Preserve the explicit
`assert_eq!(tokens.number_of_source_errors(), 0)` invariant and use a named
external snapshot for the generated token dump.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 05515c55-29e0-4c03-9737-335921d0c7eb
⛔ Files ignored due to path filters (8)
crates/antlr-rust-codegen/src/generator/snapshots/antlr_rust_codegen__generator__tests__lexer_lex_convenience.snapis excluded by!**/*.snapcrates/antlr-rust-g4-parser/src/generated/antlr_v4_lexer.rsis excluded by!**/generated/**crates/antlr-rust-g4-parser/src/generated/antlr_v4_parser.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_lexer.rsis excluded by!**/generated/**crates/antlr-rust-rs-parser/src/generated/rust_parser.rsis excluded by!**/generated/**crates/antlr-rust-runtime/src/xpath/generated/x_path_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_lexer.rsis excluded by!**/generated/**crates/antlr-rust-toml-parser/src/generated/toml_parser.rsis excluded by!**/generated/**
📒 Files selected for processing (7)
README.mdcrates/antlr-rust-codegen/src/generator/tests.rscrates/antlr-rust-codegen/src/lexer/render.rscrates/antlr-rust-codegen/src/lexer/render_model.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/lexer.rscrates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/multi_recognizer.rsthird_party/antlr-v4-grammar/self-hosted.sha256
Use the existing inline-Insta convention for disposable generated-project crates to pin the complete vocabulary name, channel, text, byte span, and column sequence. Keep lexer diagnostics as a separate invariant.
1 participant
Closes #242.
Summary
lex(input, constructor)andlex_stream(stream, constructor)in every lexer moduleCommonTokenStreamso token views retain their canonical store, source diagnostics, hidden/custom channels, and EOFByteStream, default/hidden/custom channels, EOF, and skipped-token behaviorCompatibility
The generated helpers use
InputStream,CharStream,TokenSource, andCommonTokenStream::new, all already present in generated-code API revision 6.This is an additive generated surface and does not change the generated-source
runtime contract, so the API revision remains 6.
An owned iterator cannot yield borrowing
TokenViewvalues from its own tokenstore. Returning
CommonTokenStreamkeeps ownership explicit while preservingthe concise call:
The checked-in regeneration also advances stale
0.28.0/0.30.0generatorbanners to the workspace's current
0.31.0; their generated-code API revisionremains 6.
Validation
cargo fmt --all -- --checkcargo clippy --locked --workspace --all-targets --all-features -- -D warningscargo test --locked --workspace --all-featurescargo +1.95 check --locked --workspace --all-targets --all-featuresRUSTDOCFLAGS='-D warnings -A rustdoc::private-intra-doc-links' cargo doc --locked --workspace --all-features --no-depstools/rust-syntax/update-generated.sh --checktools/toml-syntax/update-generated.sh --checktools/grammar-frontend/update-stage0.sh --update(Stage 1 and Stage 2 byte-identical; frontend corpus tests passed)shasum -a 256 -c third_party/antlr-v4-grammar/self-hosted.sha256Summary by CodeRabbit
New Features
lexandlex_streamhelpers for eagerly buffering tokens.Tests