IPO-006..010: factor scoring, dashboard, screener orchestration, enrichment, and the fail-closed AI extraction agent

[DoRmAmMu1997]

Summary

Delivers the already-combined IPO-006 through IPO-010 scope: deterministic
factor scoring and binary verdicts, the read-only dashboard, idempotent
orchestration, advisory SerpAPI enrichment, and contained AI-assisted PDF
extraction proposals behind human review.

Repository-convention waiver: this PR intentionally keeps IPO-006 through
IPO-010 together because the implementation was already delivered as one
dependent branch before remediation. This is an explicit exception to the
normal one-ticket-per-PR rule.

The central invariant is unchanged: only approved official/manual evidence
can enter scoring. AI output is a pending proposal, and web results are
quarantined advisory observations.

Post-review hardening

• Runs PDF parsing in a short-lived, killable worker with wall, page, table,
  row, column, cell, glyph, text, serialization, and Linux address-space
  budgets. The dependency-light backend/ipo_pdf_worker.py target applies the
  512 MiB Linux limit before importing pdfplumber.
• Requires citation-bound v2 financial facts: exact finite Decimal, semantic
  field/peer label, fiscal period header, cited unit, document SHA, page, and
  table cell/text span. Exact cited narrative evidence is required for objects
  of issue; submission and approval both re-resolve receipts from verified
  cached bytes.
• Makes proposal approval atomic, detects stale source bytes, enforces one
  pending proposal and semantic proposal/evaluation uniqueness in the
  database, retains reviewed proposal history, and adds --force-extract
  without allowing duplicate pending or identical proposals.
• Applies per-item enrichment quarantine, typed/versioned advisory authority,
  negation-aware red-flag observations, same-field/same-clause GMP parsing,
  canonical semantic deduplication, and streamed 1 MiB/2,000-character provider
  bounds. Web evidence cannot independently create a hard veto.
• Scores one immutable semantic evidence snapshot, persists exactly seven
  contribution rows whose sum is the score, closes concurrent evaluation
  races, and exposes source/freshness/staleness on the dashboard.
• Uses proposition-aware cited debt-purpose evidence; negated, prohibited,
  conflicting, ambiguous, missing, or legacy text cannot clear the high-debt
  caution.
• Neutralizes CommonMark controls at every untrusted IPO Streamlit
  label/caption/warning/error/expander sink without enabling unsafe HTML.

Documentation pass

• Audited all 41 Python files changed by the post-review hardening commits.
• Added or expanded beginner-friendly Google-style docstrings and detailed
  inline rationale in 24 Python files. The remaining touched modules and
  tests already had explanatory module notes or explicit regression-contract
  documentation, so they were reviewed without unnecessary churn.
• Explains process containment, citation binding, evidence authority,
  transaction/session ownership, semantic idempotency, deterministic scoring,
  migration safety, test intent, and no-network/no-unsafe-Markdown UI
  boundaries next to the code that enforces them.
• Corrects the repository documentation for the database-enforced
  one-pending-proposal invariant.
• Syntax-tree comparisons with docstrings removed confirm both documentation
  commits change documentation only; they do not alter executable Python
  behavior.

Migrations

• 20260713ipo006_screener_artifacts.py introduces the IPO-006 evaluation,
  proposal, and enrichment artifacts.
• 20260718ipo010_hardening.py adds versioned evidence/breakdown fields,
  pending-proposal and semantic-fingerprint uniqueness, non-null evaluation
  fingerprint uniqueness, retention-safe reviewed proposal references, and
  enrichment semantic first/last-seen identity.
• Alembic upgrade-to-head parity passes against Base.metadata; policy
  fixtures and hardcoded table/index expectations are updated.
• No runtime dependency or constraints change.

Security findings addressed

The original review's eight findings and the finalized scan's seven residual
findings are covered by permanent regressions: official/manual authority over
SerpAPI, exact numeric/unit/period/field binding, per-item quarantine,
pre-boundary PDF resource containment, negated debt purpose, same-clause GMP,
semantic enrichment deduplication, provider response bounds, and Markdown-safe
rendering.

The accepted residual risk remains documented: Windows has no standard-library
hard RSS limit, so it uses wall/object/text/result containment; Linux enforces
the 512 MiB child address-space limit.

Verification

• Documentation-only syntax-tree comparison: 24 Python files verified.
• Focused IPO/documentation suite: 520 passed, 1 skipped.
• Local Python 3.13 full suite: 1780 passed, 1 skipped, 89.74% coverage
  (required: 87%).
• Whole-tree mypy: 245 source files, no issues.
• Compileall, Ruff, Bandit, all pre-commit hooks, and pip-audit: passed;
  pip-audit reports no known vulnerabilities.
• Hosted Actions run
  30202559559
  on final head 594583a:
  • Python 3.11 quality and supply-chain checks: passed.
  • Python 3.12 quality and supply-chain checks: passed.
  • Docker image build: passed.

Co-authored by Claude Fable and Codex.

[DoRmAmMu1997]

Self-review: code review + security review (author-run, per AGENTS.md workflow)

Code review — verdict: no blocking findings

Verified beyond the gate suite (1640 tests, 89.9% coverage, ruff/mypy/compileall/bandit/pip-audit all green):

• Verdict precedence (missing-critical > triggered flags > score bands) and the binary invariant are pinned by tests, including a triggered flag overriding a 95-point score.
• Number-verification normalization exercised directly against edge inputs: 1,234.50 with commas/₹/Rs noise matches, parenthesised negatives (56.70) match -56.70, rounded variants match, absent numbers reject. Whitespace-stripping can merge adjacent digit runs, so a false acceptance is theoretically possible for a value that already appears on the page — the fail-closed direction is unaffected and human review remains the gate.
• Import-cycle hazard between scoring.service and the repository is resolved structurally (service deliberately not re-exported from backend.ipo.scoring; comment documents why).
• Fingerprint idempotency hashes time-derived facts (usable GMP set, near-close window) rather than the clock; clock-independence is pinned by a test.
• Known accepted nits: dashboard builder does per-issue reads (documented; universe is dozens of issues, page cached); the entirely_ofs_weak_growth evidence line reads slightly oddly for a zero-fresh/zero-OFS issue (status is still correct); approve/approve race leaves one extra append-only revision and fails loudly (documented in the docstring and error message).

Security review — verdict: no critical/high findings; residual risks documented

• SSRF: no new arbitrary-URL fetches. Enrichment reuses the fixed-endpoint SerpAPI client (result links stored as data, never fetched); the extraction agent reads only the hash-verified local cache and never sees a path.
• Prompt injection (TEST-003): every prospectus excerpt handed to the model and every web snippet stored passes the shared quarantine engine first. Hits are non-retryable (extractor) or replace text with the blocked marker (quarantined=true rows); raw hostile text never reaches storage, logs, or the model.
• Trust boundaries are structural: AI output persists only as pending proposals scoring never reads; approval re-validates the payload through the strict manual contract and re-hashes the cached PDF bytes; enrichment signals are typed away from every financial-statement field.
• AuthZ: re-score requires MANAGE_IPO_DATA at the dispatch boundary; proposal review lives inside the admin page with its defense-in-depth re-check; reviewer identity comes from the server session, never the form; review decisions and re-scores are audit-logged.
• Secret safety: all new persisted JSON passes normalize_secret_safe_json; rejection notes pass redact_text; failure paths carry exception type names and stable codes only.
• Resource exhaustion: hostile-PDF caps (800 pages / 20k chars / 20 tables / 200-char cells), bounded section chunks, bounded search results, linear regexes.
• Residual risks (accepted, consistent with the audit register's posture): (1) model-transcribed free text (objects of issue, peer names, verifier notes) is displayed to the reviewing admin — a hostile document could try to social-engineer the reviewer through it; the control is review against the source PDF plus host-verified citations. (2) The digit-merge false-acceptance noted above. Neither creates a path for unverified data to reach scoring.

🤖 Generated with Claude Code

[DoRmAmMu1997]

Codex review — changes required before merge

I reviewed the exact PR range de8f199…aab2cd4, including the IPO-006 through IPO-010 implementation, focused runtime reproductions, migration/layering policy, and the rendered dashboard path. The current CI is green, but the review found eight validated security/data-integrity issues:

1. Medium — SerpAPI authority inversion: uncorroborated litigation snippet text can become a score-independent hard veto, including negated text such as “No litigation…”.
2. Low — numeric evidence equivalence: rounded values, digit substrings, and values assembled across cells can pass host verification.
3. Low — uncited unit scale: model-selected units are allowlisted but not bound to cited source evidence before HIGH.
4. Low — row-wide quarantine fail-open: one hostile result quarantines clean siblings, after which the caution path reports NOT_TRIGGERED.
5. Low — PDF page limit is post-materialization: the complete page tree is built before the page cap is checked.
6. Low — PDF table limits are post-extraction/incomplete: all tables are extracted before slicing, with no row/column/total-cell budget.
7. Low — PDF text limit is post-extraction: the parser constructs the complete text map/string before truncation.
8. Low — debt-purpose negation: free-text substring matching treats negated repayment wording as affirmative evidence and can suppress the high-debt caution.

The broader correctness pass also found work required around proposal source freshness and atomic approval, database-enforced pending/evaluation uniqueness, reviewed-proposal retention, extraction rerun idempotency, semantic score fingerprints, consistent input snapshots, period ordering, page-safe chunks, continuation-page classification, GMP proximity parsing, dashboard provenance/freshness, factor-impact ordering, Markdown-safe rendering, and stale architecture/operations documentation.

The implementation will keep this already-combined IPO-006…010 PR, with an explicit waiver of the repository’s normal one-ticket-per-PR convention. Fixes will be test-first, preserve official/manual evidence precedence and human approval, and retain the existing deterministic scoring weights and verdict bands.

This is a COMMENT review rather than an approval/request-changes event because the automation identity is also the PR author.

[DoRmAmMu1997]

Codex post-security review — changes still required

The finalized branch-diff scan closed 24/24 coverage rows and validated seven remaining issues against local remediation head 7130cc3a606f0ddb9a2b3e30d5031a4df026737a (the six local remediation commits are not pushed yet):

• Financial evidence is token-bound but not yet semantically bound to the claimed field, period, and unit context.
• A page-wide unit fallback can select unrelated text instead of the cited value's own table/text context.
• Negated debt-repayment language can suppress the high-leverage caution.
• GMP parsing can consume an unrelated percentage from a neighboring clause.
• Semantically identical enrichment observations can be persisted repeatedly and overweighted.
• Untrusted Markdown control characters are not neutralized at every Streamlit rendering boundary.
• Provider-result object work is not bounded early enough before downstream processing.

These are fixable without changing the intended deterministic weights, recommendation bands, binary verdict, or advisory-only web-evidence policy. I will add failing regression tests first, apply the narrow authority/resource-boundary fixes, rerun the original local checks plus the repository gates, then push and update this review with evidence.

Repository convention waiver: PR #108 intentionally retains IPO-006 through IPO-010 as the already-combined delivery; this is an explicit exception to the normal one-ticket-per-PR rule.

[DoRmAmMu1997]

Codex completion review — ready to merge

I re-reviewed the complete range
de8f199b937a2ec9c7f78d1ed86bf6924841fac4...326061d838dab57e6a17e0ad147582e0b88c44c8
after remediation.

The original eight validated findings and the seven residual post-review
findings are addressed with permanent regression coverage:

• official/manual evidence authority prevents SerpAPI hard-veto inversion;
• PDF work is killable and bounded before crossing the worker boundary;
• financial values, units, periods, field/peer meaning, pages, cells/spans,
  source tokens, and document SHA are atomic v2 evidence;
• cited narrative evidence and proposition-aware debt-purpose classification
  fail closed on negation, ambiguity, conflict, legacy data, and forgery;
• proposal approval, retention, uniqueness, force-rerun, and evaluation
  idempotency are transactionally and semantically enforced;
• per-item quarantine, same-clause GMP, semantic deduplication, advisory
  authority, and streamed provider limits prevent web-evidence escalation;
• seven-row score provenance, source freshness/staleness, deterministic impact
  ordering, and Markdown-safe Streamlit rendering are public and tested.

The first hosted run exposed a Linux-only spawn bootstrap problem: importing
the broad backend.ipo facade consumed address space before the 512 MiB child
limit. Commit 326061d moves the target to dependency-light
backend/ipo_pdf_worker.py, preserving the limit and allowing pdfplumber to
start. The rerun is fully green:

• Python 3.11 quality and supply-chain checks: passed.
• Python 3.12 quality and supply-chain checks: passed.
• Docker image build: passed.
• Focused IPO suite: 464 passed, 1 skipped.
• Exact cross-finding selection: 68 passed.
• Migration parity: 13 passed.
• Whole-tree mypy: 245 source files, no issues.
• Ruff, compileall, Bandit, pre-commit config, and pip-audit: passed.

The PR description now includes the explicit IPO-006…010 combined-ticket
waiver, both migration notes, security closure, residual Windows memory-limit
risk, and verification evidence. The live PR is MERGEABLE / CLEAN, all
checks pass, and there are no inline review threads.

This is a COMMENT review because the automation identity is also the PR
author and cannot self-approve.

[DoRmAmMu1997]

Documentation pass completed on final head 594583a.

• Audited all 41 Python files touched by the post-review hardening commits.
• Added or expanded beginner-friendly docstrings and inline rationale in 24
  files; the other touched modules and tests already contained explanatory
  module or regression-contract documentation.
• Covered the PDF worker boundary, typed/cited evidence verification,
  enrichment authority, transaction ownership, semantic idempotency,
  deterministic scoring, migration invariants, test intent, and Streamlit
  safety rules.
• Verified both commits are documentation-only by comparing Python syntax
  trees after removing docstrings.
• Local: 1780 passed, 1 skipped, 89.74% coverage; all pre-commit hooks, Ruff,
  mypy, compileall, Bandit, and pip-audit passed.
• Hosted run 30202559559: Python 3.11, Python 3.12, and Docker all passed.

No review threads remain unresolved, and the PR is mergeable.