feat(agent): multi-section configs, local payloads, GEPA optimize, reflexion/judge gates - #62
Merged
Merged
Conversation
…flexion/judge gates
- Multi-section configs: one {template, sections} config per paper; each section
owns its own source/url/sheet. validate_table_config gates every section and
map_coverage reports an aggregate (overall/min/per-section). New GENE_DISEASE fixture.
- --local/-l: run the pipeline on a local payload (one DIR for all ids, or PMCid=DIR
mappings) instead of fetching from PMC-AWS; fails loud (exit 2) on a missing dir.
- --optimize/-o: first-class dspy.GEPA prompt optimization that persists optimized
instructions (--instructions-out), reloadable via --instructions-file, bounded by
--max-metric-calls over a --dataset.
- --reflexion: tier-2 LLM reflexion improver for when the deterministic proposer stalls.
- --judge-model/--judge-threshold: semantic judge gate; MAPPED additionally requires the
score to clear the threshold (0.5 when unset).
- PDF main-text context via pdfminer.six; BUILT_UNMEASURED terminal non-failure; multi-cwd
resolution of relative source.local.
- docs: complete the cli.md SSOT flag table and agent.md depth for all new flags (fixes
the docs-coverage guardrails).
655 passed, 29 skipped; ruff + ruff-format + pyright clean.
|
Warning Review limit reached
Next review available in: 17 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe agent now supports multi-section table configurations, PDF text extraction, local payloads, per-section coverage, reflexion and semantic gates, ChangesAgent workflow expansion
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant run_supervisor
participant propose_config_candidates
participant build_and_audit
participant judge_model
CLI->>run_supervisor: pass workflow options
run_supervisor->>propose_config_candidates: generate ranked candidates
run_supervisor->>build_and_audit: run candidate build
run_supervisor->>judge_model: request optional semantic score
judge_model-->>run_supervisor: return gate decision
run_supervisor-->>CLI: return final status and metrics
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
tests/test_agent_supervisor.py (1)
444-459: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd
head: bool = Falsetofake_build.The improve loop is not gated by
measured=False. If candidate generation returns an edit, it callsbuild_and_audit(..., head=True), which this stub cannot accept and which would mark the recordSKIPPED.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_agent_supervisor.py` around lines 444 - 459, Add the optional head: bool = False parameter to the fake_build test stub, preserving its existing return behavior so calls from the improve loop with head=True are accepted and do not produce a skipped record.src/tablassert/agent.py (1)
1571-1663: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the shared coverage-entry lookup and heuristic bodies.
_propose_categoryrepeats thecolumns_forclosure from_propose_multi_section(Lines 1478-1489) verbatim, and_apply_category_to_noderepeats the taxonomic/noise/exclude branches of_edit_node(Lines 1407-1429). A future change to one heuristic must be mirrored in two places.Extract a module-level
_columns_selector(report)helper and drive_edit_nodefrom the category functions, so the full edit becomes "all categories plus the chemical fallback".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/tablassert/agent.py` around lines 1571 - 1663, Extract the duplicated coverage lookup into a module-level _columns_selector(report) helper and use it from _propose_category and _propose_multi_section. Refactor _edit_node to apply the shared taxonomic, noise, and exclude category logic through _apply_category_to_node, leaving only the chemical fallback as full-edit-specific behavior. Preserve existing category ordering, rationale, and multi-section coverage behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/tablassert/agent.py`:
- Around line 2614-2634: Validate full-build results before committing them in
the improve loop: in src/tablassert/agent.py lines 2614-2634, only accept and
assign the candidate when full_report["ok"] is true and its coverage is strictly
greater than the prior current_cov; otherwise continue to the next candidate.
Apply the same guard to full_report3 in src/tablassert/agent.py lines 2637-2658,
preserving current_config, coverage history, and best coverage when validation
fails.
- Around line 3229-3241: Update the file-reading logic around the YAML-loading
function so both p.read_text() and yaml.safe_load() execute inside the try
block. Catch OSError, UnicodeDecodeError, and yaml.YAMLError, returning None for
any unreadable file or invalid YAML while preserving the existing
parsed-instructions handling.
In `@src/tablassert/cli.py`:
- Around line 655-663: Check the stats error field on the result returned by
agent_mod.run_gepa before deriving or saving optimized instructions. When a GEPA
compilation error is present, avoid save_optimized_instructions and the success
message, report the failure, and return a non-zero CLI status; preserve the
existing save-and-success flow for successful results.
- Around line 635-644: Update the --local parsing loop over specs to validate
both the stripped PMC id and directory string before constructing Path or adding
to mapping. Reject either blank component with the existing stderr message and
SystemExit(2), while preserving normal PMCid=DIR handling.
- Line 545: Validate judge_threshold before invoking the supervisor: reject
non-finite values and any value outside the inclusive range [0, 1], and exit
with status 2 for invalid input. Keep valid thresholds, including None, on the
existing execution path.
- Around line 652-656: Update the optimize flow around agent_mod.make_dspy_lm
and agent_mod.run_gepa to pass the selected --backend value into the GEPA model
configuration. Ensure the constructed reflection LM uses the requested backend
rather than always defaulting to the OpenAI provider, while preserving the
existing resolved model ID, base, and key behavior.
---
Nitpick comments:
In `@src/tablassert/agent.py`:
- Around line 1571-1663: Extract the duplicated coverage lookup into a
module-level _columns_selector(report) helper and use it from _propose_category
and _propose_multi_section. Refactor _edit_node to apply the shared taxonomic,
noise, and exclude category logic through _apply_category_to_node, leaving only
the chemical fallback as full-edit-specific behavior. Preserve existing category
ordering, rationale, and multi-section coverage behavior.
In `@tests/test_agent_supervisor.py`:
- Around line 444-459: Add the optional head: bool = False parameter to the
fake_build test stub, preserving its existing return behavior so calls from the
improve loop with head=True are accepted and do not produce a skipped record.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e46fc458-f4b8-45ae-a71b-c4254ca8732d
⛔ Files ignored due to path filters (2)
tests/agent_fixtures/GENE_DISEASE/source_table.csvis excluded by!**/*.csvuv.lockis excluded by!**/*.lock
📒 Files selected for processing (16)
docs/agent.mddocs/cli.mdpyproject.tomlsrc/tablassert/agent.pysrc/tablassert/cli.pytests/agent_fixtures/GENE_DISEASE/reference_config.yamltests/test_agent_assembly.pytests/test_agent_build.pytests/test_agent_cli.pytests/test_agent_context.pytests/test_agent_coverage.pytests/test_agent_eval.pytests/test_agent_multisection.pytests/test_agent_propose.pytests/test_agent_supervisor.pytests/test_cover_agent_propose.py
…on, GEPA optimize - Improve loop: commit a confirming full build IFF it succeeded (ok) AND its coverage is strictly greater than the prior best; a failing or lower-scoring full build (optimistic 5-row head sample) no longer regresses current_config, the monotonic coverage_history, or best_coverage (tier 1 + tier 2). - load_optimized_instructions: read inside the try with explicit utf-8 and catch OSError/UnicodeDecodeError too, so an unreadable instructions file returns None instead of aborting the run. - make_dspy_lm: honor --backend (openai/ prefix vs litellm pass-through); the --optimize path now forwards --backend instead of always routing through openai. - cli agent: reject --judge-threshold outside [0,1] or non-finite (exit 2); reject --local PMCid=DIR with a blank id or dir (exit 2) so PMC1= is not the cwd; --optimize exits 1 without saving/reporting success when GEPA stats has an error. - Extract _columns_selector to dedup the identical columns_for closure shared by the multi-section and per-category proposers (behavior-preserving). - Tests: tier-1/tier-2 full-build rejection, judge-threshold/--local/--optimize validation, GEPA-error exit, make_dspy_lm backend, unreadable instructions; add head=False to a fake_build stub.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expands the autonomous
tablassert agentpipeline from one-section-per-paper to a richer, still-deterministic supervisor: multi-section configs, local (non-open-access) payloads, first-class GEPA prompt optimization, and opt-in reflexion/judge gates — with thedocs/cli.mdflag table brought back in sync.Multi-section configs (one per paper)
{template, sections}—templatecarries only shared provenance (repo+publication, nosource); each entry insectionsowns its ownsource(localpath andsource.url, plussheet/row_slice/delimiter) andstatement, so one paper maps every supplementary table/worksheet.validate_table_configreplacesvalidate_sectionas the final-answer gate and validates every section; a config is accepted only when all sections are schema-valid.map_coveragemeasures each section (_measure_section) and reports an aggregate —overall(mean),min(weakest),measured(true iff all measured), plus a per-section breakdown;propose_config_editedits each section from its own coverage entry (_propose_multi_section).tests/agent_fixtures/GENE_DISEASE/(a gene~disease config in multi-section shape with PMID provenance) to keep the offline heuristic judge and validation honest on a distinct config; covered by the newtests/test_agent_multisection.py(+340).Local payloads (
--local/-l)--localruns the same derive/build/improve pipeline on an article held locally (e.g. a non-open-access paper).DIRapplied to every id, or per-articlePMCid=DIRmappings (parse_localincli.py,_resolve_local_dirinagent.py). When set, the supervisor locates files locally and skips the PMC-AWS fetch; a missing dir fails loud (exit 2).GEPA prompt optimization (
--optimize/-o)tablassert agent --optimizerunsdspy.GEPAwith a real reflection LM (make_dspy_lm) and persists the optimized instructions (save_optimized_instructions) instead of running the supervisor; reload via--instructions-file.--instructions-out(default<state-dir>/optimized_instructions.yaml),--max-metric-calls(budget, default8), and--dataset(YAML/JSON{table_summary, coverage_feedback}examples viaload_gepa_dataset).Reflexion improver & semantic judge (opt-in)
--reflexion: a tier-2 LLM reflexion improver (llm_propose_config_edit,make_prompt_callable) for edits that may change predicate/source when the deterministic proposer stalls — same model config, built lazily.--judge-model/--judge-threshold: a semantic judge scores the output; when--judge-modelis set,MAPPEDadditionally requires the normalized score to clear--judge-threshold(0.5when unset). Without it, the coverage gate alone decides.Robustness
pdfminer.six(new[agent]dep) extracts a.pdfmain text into data-fenced context (_extract_pdf_text), so PDF-only articles still give the agent main-text context.BUILT_UNMEASURED: a new terminal non-failure for a config that builds but whose fullmap coverage can't be measured (unreproducible source frame) — neitherMAPPEDnorSKIPPED; the best config is still written and reusable.source.localis resolved against the build workdir as well as the cwd (_candidate_cwds) before a config is declared unmeasurable.Design
--optimize,--reflexion, and--judge-modelruns need a live model; the offline suite exercises these paths via injectable stubs (gepa_cls, monkeypatchedrun_gepa/make_dspy_lm), so no network fires in CI.Docs
docs/cli.md— completed the SSOTagentflag table with all nine new flags (this is what thetests/test_docs_cli_coverage.pyguardrails enforce).docs/agent.md— multi-section model,--local,--optimize, and a new "Optional gates: reflexion improver & semantic judge" subsection;pdfminer.sixpin documented.Testing
uv run pytest -q→655 passed, 29 skipped(skips are live-model/network paths); the 3 previously-failingtest_docs_cli_coverage.pyguardrails now pass.uv run ruff check .→All checks passed!uv run ruff format --check .→64 files already formatteduv run pyright→0 errors, 0 warnings, 0 informationsPassedon commit.Questions for the reviewer
--judge-thresholddefaults toNonein the CLI and falls back to0.5in code — is0.5the right floor, or should the threshold be required when--judge-modelis set?{template, sections}) is now the only authored shape; is a migration path for existing single-section configs needed, or are there no configs in the wild yet?Summary by CodeRabbit
New Features
Documentation