docs(module-b): commit Module A/C contracts + runbook, synced to current code

.gitignore

@@ -69,6 +69,9 @@ standards_cache.sqlite
 !AGENTS.md
 !docs/faq.md
 !docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md
+!docs/gsoc_2026_module_b/module_a_contract.md
+!docs/gsoc_2026_module_b/module_c_contract.md
+!docs/gsoc_2026_module_b/module_b_runbook.md
 
 ### Dev DBDumps
 *.sql

docs/gsoc_2026_module_b/module_a_contract.md

@@ -0,0 +1,211 @@
+# Module A → Module B Input Contract
+
+**Audience:** the GSoC 2026 contributor implementing Module A (Information Harvesting — nightly cron job that fetches content from OWASP repos and feeds). **Status:** draft **v0.4** (2026-08-16). Reconciled against the orchestrated-pipeline hand-off (DB table, not a JSONL file) and Module C's shipped consumer (`feed_item` rss locator kind). Pending review by mentors and the Module A contributor.
+
+This document specifies the format Module A emits so that Module B (Noise/Relevance Filter) can consume it. Module B is the only downstream consumer in v1; any change to this format is a breaking change for B.
+
+---
+
+## Changelog (v0.3 → v0.4)
+
+- **Delivery moved from a JSONL file to a DB table.** The orchestrated pipeline supersedes the earlier "JSONL file via `cre.py --filter_changes <path>`" transport. Module A now writes each record as a row in the shared Postgres `harvest_input` table (JSONB `payload` + top-level `pipeline_run_id` + `status`); Module B reads the pending rows for a run. The `--filter_changes` CLI was never built; Module B's entry point is `cre.py --run_noise_filter --run_id <id>`.
+- **rss `locator.kind` reserved value is `feed_item`** (was `feed_post`), aligning with Module C's shipped consumer (PR #1011), which addresses rss rows by `locator_kind = "feed_item"`. github rows are unchanged (`repo_path`). rss is still not emitted; this only fixes the reserved name so all three modules agree.
+
+## Changelog (v0.2 → v0.3)
+
+Driven by reconciliation with Module A's actual mock output. The shape is significantly different from what the v0.2 draft anticipated.
+
+- **Structural shift:** record fields are no longer flat. Three nested objects now exist: `source` (provenance), `span` (chunk position within parent artifact), `locator` (addressable identity).
+- **New top-level fields:** `schema_version`, `chunk_id`, `pipeline_run_id`.
+- **Renames / relocations:**
+  - `chunk_text` → `text`
+  - `source_type` → `source.type`
+  - `repo` → `source.repo`
+  - `commit_sha` → `source.commit_sha`
+  - `author_date` → `source.committed_at`
+  - `file_path` → `locator.path` (also `locator.id` mirrors for now)
+  - `chunk_index` → `span.index`
+- **New `span` payload:** beyond `index` (which `chunk_index` already provided), `span` also carries `total`, `heading_path` (the markdown heading breadcrumb), `start_char_idx`/`end_char_idx`, `start_line`/`end_line`.
+- **`content_hash` removed.** Module A does not emit a content hash. **Module B computes its own** by applying the v0.2 normalization rules (NFC, line endings, whitespace, code-fence preservation) and SHA-256-ing the result. Used by B as the `knowledge_queue` dedup key.
+- **`commit_message` removed.** Module A does not emit commit messages. Module B's LLM prompt now uses `span.heading_path` as the semantic context signal instead (e.g. `["Authentication", "JWT"]` is a richer disambiguator than a commit message).
+- **`source.type = "rss"` is reserved.** Mock data is github-only; the discriminated-union schema accepts RSS shape so we're ready when Module A emits feed records.
+- **Mock note:** the mock data uses placeholder values like `"abc123"` for `commit_sha` and `"…"` (encoded as corrupted UTF-8 `â¦` in the source) for path segments inside `chunk_id`. Production must emit real 40-char SHAs and clean UTF-8.
+
+---
+
+## Transport
+
+- **Format:** one JSON object per chunk — the record described below — stored verbatim as the JSONB `payload` of a `harvest_input` row. (Module B's local test fixtures still keep the same records as JSONL; the on-the-wire shape of a single record is identical either way.)
+- **Delivery:** Module A writes each record as a row in the shared Postgres **`harvest_input`** table — columns `payload` (JSONB, the record), `pipeline_run_id` (top-level, run-scoping), `status` (`pending` → set by A; `processed`/`error` → set by B), plus `id`/`created_at`. Module B reads that run's pending rows (`cre.py --run_noise_filter --run_id <id>`), classifies, and writes keepers to `knowledge_queue`. See `module_b_runbook.md` for the operational how-to. *(The earlier "JSONL file via `cre.py --filter_changes`" delivery is superseded and was never implemented.)*
+- **Future (out of scope for v1):** object-storage URL (S3/MinIO) for large or out-of-band payloads.
+- **Record size:** governed by Module A's chunking config (default `max_chars=4000`). Module B truncates internally at 1500 chars before sending to the LLM.
+
+## Required fields (top level)
+
+| Field | Type | Constraints |
+|---|---|---|
+| `schema_version` | string | E.g. `"0.2.0"`. Pinned by Module A; B reads but does not validate version semantics. |
+| `chunk_id` | string | Module-A-stable identifier. Format observed in mock: `chk:<artifact_id>:<chunk_index>` (i.e. `chk:art:OWASP/<repo>:<path>:<idx>`). |
+| `artifact_id` | string | Identifier for the parent document. Format observed in mock: `art:<repo>:<path>`. Stable across chunks of the same artifact. |
+| `pipeline_run_id` | string | Identifier for the Module A pipeline execution that produced this record. E.g. `"20260201T020000Z"`. Used by B for audit / replay grouping. |
+| `text` | string | The normalized chunk content. Markdown markers (`#`, `**`, code fences) preserved; HTML stripped; whitespace collapsed in prose but preserved inside code fences (`` ``` ``…`` ``` ``) and `<pre>` blocks. |
+| `span` | object | Position metadata. See below. |
+| `source` | object | Provenance discriminator (`source.type`). See below. |
+| `locator` | object | Addressable identity. See below. |
+
+## Required `span` payload
+
+| Field | Type | Constraints |
+|---|---|---|
+| `index` | int ≥ 0 | Zero-based chunk index within the parent artifact. |
+| `total` | int ≥ 1 | Total chunks Module A produced from this artifact. |
+| `heading_path` | array of strings | Breadcrumb of enclosing markdown headings (e.g. `["Authentication", "JWT"]`). Empty array if the chunk precedes all headings. Used as a semantic signal in B's LLM prompt. |
+| `start_char_idx` | int ≥ 0 (optional) | Character index of chunk start in the normalized artifact text. |
+| `end_char_idx` | int ≥ 0 (optional) | Character index of chunk end (exclusive). |
+| `start_line` | int ≥ 0 (optional) | 1-based line number of chunk start in the normalized artifact. |
+| `end_line` | int ≥ 0 (optional) | 1-based line number of chunk end. |
+
+## Required `source` payload — discriminated union
+
+`source.type` discriminates between provenance shapes.
+
+**When `source.type = "github"`:**
+
+| Field | Type | Constraints |
+|---|---|---|
+| `type` | string | Literal `"github"`. |
+| `repo` | string | Format `^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$`. Allows OWASP dots/dashes. |
+| `commit_sha` | string | Production: 40-char hex SHA-1. Mock: 6+ char placeholders accepted. |
+| `committed_at` | string | ISO-8601 timestamp. |
+
+**When `source.type = "rss"` (reserved, not yet emitted):**
+
+| Field | Type | Constraints |
+|---|---|---|
+| `type` | string | Literal `"rss"`. |
+| `feed_url` | string | The canonical feed URL Module A subscribed to. |
+| `post_guid` | string | The `<guid>` or `<id>` of the post within the feed. |
+| `post_published_at` | string (optional) | ISO-8601 timestamp. |
+
+## Required `locator` payload
+
+| Field | Type | Constraints |
+|---|---|---|
+| `kind` | string | Scheme. `"repo_path"` for github sources today. Reserved: `"feed_item"` for RSS (matches Module C's consumer, PR #1011). |
+| `id` | string | Unique identity within the scheme. For `repo_path`: the file path. |
+| `path` | string | Convenience duplicate of `id` for `repo_path` (`id == path`). |
+
+## Example record (github source — mock-shaped)
+
+```json
+{
+  "schema_version": "0.2.0",
+  "chunk_id": "chk:art:OWASP/ASVS:4.0/en/0x12-V3-Authentication.md:0",
+  "artifact_id": "art:OWASP/ASVS:4.0/en/0x12-V3-Authentication.md",
+  "pipeline_run_id": "20260201T020000Z",
+  "text": "Authentication should use MFA",
+  "span": {
+    "index": 0,
+    "total": 3,
+    "heading_path": ["Authentication", "JWT"],
+    "start_char_idx": 0,
+    "end_char_idx": 98,
+    "start_line": 10,
+    "end_line": 12
+  },
+  "source": {
+    "type": "github",
+    "repo": "OWASP/ASVS",
+    "commit_sha": "abc123",
+    "committed_at": "2026-02-01T01:00:00Z"
+  },
+  "locator": {
+    "kind": "repo_path",
+    "id": "4.0/en/0x12-V3-Authentication.md",
+    "path": "4.0/en/0x12-V3-Authentication.md"
+  }
+}
+```
+
+## Content hashing — Module B's responsibility
+
+Module A does **not** emit `content_hash`. Module B computes one on ingest in `application/utils/noise_filter/hashing.py`:
+
+1. Apply v0.2 normalization rules to `text`:
+   - Unicode NFC
+   - CRLF / CR → LF
+   - Trailing whitespace per line stripped
+   - Leading/trailing blank lines stripped
+   - Runs of spaces/tabs in prose collapsed to one space
+   - Whitespace inside fenced code blocks (`` ``` `` … `` ``` ``) and `<pre>` blocks preserved verbatim
+2. `content_hash = hashlib.sha256(normalized_text.encode("utf-8")).hexdigest()`
+
+The hash becomes the `UniqueConstraint` key on `knowledge_queue` — re-feeding identical normalized content (e.g. mirrored docs, replayed pipeline runs) collapses to one queue row.
+
+**Future:** if Module A starts emitting `content_hash`, B switches via `CRE_NOISE_FILTER_TRUST_A_HASH=true` to skip recomputation. Until then, B is self-sufficient.
+
+## Normalization ownership
+
+- **Module A owns "normalize for chunking":** semantic chunking, heading-path tracking, char/line offsets.
+- **Module B owns "normalize for hashing" + "defensive sanitization":** the rules above (for `content_hash`), plus TRACT-style sanitize.py (zero-width chars, PDF ligatures, hyphenation rejoin) as Stage 1.5 of B's pipeline.
+
+If Module A's own normalization differs from B's (e.g. A doesn't collapse prose whitespace), B's hash still works — B always normalizes before hashing. The two normalizations don't need to agree.
+
+## Path-filtering ownership
+
+- **Module A owns coarse exclusion:** the `paths_exclude` globs in its source config (e.g. `["**/package-lock.json", "**/CNAME"]`). These never reach B.
+- **Module B owns fine-grained noise filtering:** `application/utils/noise_filter/noise_patterns.yaml`. Catches things A's source config didn't anticipate.
+
+If both modules block the same path, that's fine — B silently no-ops. The two lists drift independently.
+
+## Stability guarantees
+
+- **Module B reads only the fields listed above** (and ignores everything else). Extra fields Module A adds — `pr_number`, `author`, `tags`, `supersedes_artifact_id`, etc. — are silently accepted (`extra="ignore"` on B's Pydantic models). Safe to extend.
+- **Renaming or removing any required field is breaking.** Requires a contract version bump and coordinated changes in `application/utils/noise_filter/schemas.py`.
+- **Changing the semantics of a field is breaking** even if the name stays the same. Example: switching `locator.path` to absolute paths would break B's regex pre-filter.
+
+## Idempotency on Module B's side
+
+Module B's `knowledge_queue` uses `UniqueConstraint(content_hash)` as the dedup key. Consequences:
+- Re-feeding the same normalized content is a no-op.
+- The same content reaching B via two different sources collapses to one row.
+- To force re-classification (e.g. prompt changed), Module B will provide an `--allow_duplicate_hash` flag on the CLI. Out of scope for v1.
+
+## Versioning
+
+This contract is **v0.3** (draft). When ratified, becomes v1.0. semver applies:
+- v1.X = additive, non-breaking field additions.
+- v2.0 = breaking changes.
+
+The version applies to *this contract*, not to Module A's release cadence.
+
+## JSON Schema artifact
+
+The machine-readable schema is generated from Module B's Pydantic models and committed at:
+
+```
+docs/gsoc_2026_module_b/module_a_contract.schema.json
+```
+
+Both modules SHOULD validate against it in CI. Module B's Pydantic model (`application/utils/noise_filter/schemas.ChangeRecord`) is the canonical source; the JSON Schema file is derived via `ChangeRecord.model_json_schema()`.
+
+## Test fixtures
+
+Module B keeps fixtures at:
+
+```
+application/tests/noise_filter/fixtures/
+├── module_a_mock.jsonl          # Module A's 20-record mock (canonical, what A actually emits)
+├── candidate_commits.json       # B's own stand-in harvest of ~100 OWASP commits in Module A's shape
+└── labeled_data.json            # candidate_commits.json + KNOWLEDGE/NOISE/UNCERTAIN labels
+```
+
+Module A contributors are welcome to add fixtures here as PRs — small files (≤30 records each) covering edge cases (PDF-extracted text, HTML-derived chunks, RSS posts) help Module B's regex and prompt iteration.
+
+## Out-of-scope for this contract
+
+- How Module A produces these records (GitHub API vs `git log` vs webhook vs feed poll).
+- Where Module A stores raw artifacts before chunking.
+- Failure handling on the Module A side (rate-limit retries, partial commits).
+- Authentication / API keys (Module A's concern).
+- Module A's source-config schema (`schema_version`, `sources`, `chunking`) — that's internal to A.

docs/gsoc_2026_module_b/module_b_runbook.md

@@ -0,0 +1,121 @@
+# Module B — How to Run It (Orchestrator Runbook)
+
+**Audience:** whoever builds/operates the daily orchestrator. **Status:** v0.2 (2026-08-16).
+Companion to `module_c_contract.md` (the B→C contract). This is the operational *how*.
+
+Module B is a **stateless batch step**: the orchestrator invokes it once per harvest run; it reads that run's chunks from a DB table, classifies them, writes the keepers to another table, and exits with a JSON summary. It does not run continuously and does not schedule itself.
+
+---
+
+## 1. One-time setup
+
+**Tables.** Module B owns two tables, created by its Alembic migration
+`d4e5f6a7b8c9_add_module_b_tables` (part of the chain — other modules' migrations
+now chain after it, so it is no longer the head):
+- `harvest_input` — Module A writes here; B reads.
+- `knowledge_queue` — B writes here; Module C reads.
+
+Apply with:
+```bash
+FLASK_APP=cre.py FLASK_CONFIG=development flask db upgrade
+```
+A full from-empty `flask db upgrade` on Postgres runs the whole chain, creating
+Module B's tables at migration `d4e5f6a7b8c9` (now mid-chain; the current head
+moves as other modules add migrations after it — the from-empty upgrade still
+reaches B's tables regardless). The earlier `uq_pair` duplicate-index bug is
+fixed and merged. One caveat: **C's pgvector migration (`c7d8e9f0a1b2`)
+requires `CRE_EMBED_EXPECTED_DIM` to be set** on an empty DB (it can't infer the
+vector dimension with no embeddings yet) — a pre-existing requirement of that
+migration, e.g. `CRE_EMBED_EXPECTED_DIM=3072 flask db upgrade`. Postgres needs
+the `vector` extension (use the `pgvector/pgvector` image or `CREATE EXTENSION
+vector;`).
+
+**Environment variables:**
+| Var | Purpose | Default |
+|---|---|---|
+| `GEMINI_API_KEY` | LLM credential (Gemini) | — (required) |
+| `CRE_NOISE_FILTER_LLM_MODEL` | classification model | `gemini/gemini-2.5-flash-lite` |
+| `CRE_NOISE_FILTER_BATCH_SIZE` | chunks per LLM request | `10` |
+| `CRE_NOISE_FILTER_MAX_CHARS` | per-chunk truncation | `1500` |
+| `CRE_NOISE_FILTER_CONFIDENCE_THRESHOLD` | KNOWLEDGE-below → UNCERTAIN | `0.8` |
+
+Module B needs no ML libraries (no torch/sentence-transformers) — just `litellm` (in the slim prod requirements) + the Gemini key.
+
+---
+
+## 2. The input table `harvest_input` (Module A writes)
+
+One row per harvested chunk:
+| Column | Notes |
+|---|---|
+| `id` | any stable PK |
+| `pipeline_run_id` | groups a run — B reads exactly this |
+| `status` | `pending` (A sets) → `processed`/`error` (B sets) |
+| `payload` | **JSONB** — Module A's ChangeRecord v0.3 (nested `source`/`span`/`locator`), written as-is |
+| `created_at` | timestamp |
+
+B reads `WHERE pipeline_run_id = :run_id AND status = 'pending'`.
+
+---
+
+## 3. Invoking Module B (the command the orchestrator runs)
+
+```bash
+python cre.py --run_noise_filter --run_id <pipeline_run_id> --cache_file <db-url>
+```
+- `--run_id` — the run to process (**required**).
+- `--cache_file` — the database URL (e.g. `postgresql://user:pass@host:5432/opencre`).
+- `--noise_filter_dry_run` — optional; classify but write nothing / mark nothing (for testing).
+
+The process runs the gate (regex path filter → sanitize → LLM classify), writes keepers, marks the input rows, and exits.
+
+---
+
+## 4. Completion signal (how the orchestrator knows it's done)
+
+- **Exit code `0`** = success; **non-zero** = hard failure (e.g. DB unreachable) → safe to retry the same `run_id` (B is idempotent).
+- **stdout** = a one-line JSON summary:
+```json
+{"run_id":"20260201T020000Z","read":512,"parse_errors":0,"dropped_noise":172,
+ "kept_knowledge":300,"kept_uncertain":40,"inserted":338,"deduped":2,
+ "dry_run":false,"status":"ok"}
+```
+| Field | Meaning |
+|---|---|
+| `read` | input rows for the run |
+| `parse_errors` | payloads that failed validation (rows marked `error`) |
+| `dropped_noise` | dropped as NOISE (regex + LLM) — not queued |
+| `kept_knowledge` / `kept_uncertain` | classified as KNOWLEDGE / UNCERTAIN |
+| `inserted` | rows written to `knowledge_queue` |
+| `deduped` | keepers skipped as duplicate content |
+| `status` | `ok` (per-chunk errors are contained, not fatal) |
+
+---
+
+## 5. The output table `knowledge_queue` (Module C reads)
+
+B inserts `KNOWLEDGE` and `UNCERTAIN` rows (never `NOISE`), deduped on `content_hash`. Module C reads unconsumed rows and sets `consumed_at`. Full schema + read query: `module_c_contract.md` (v0.2).
+
+---
+
+## 6. Orchestrator sequencing
+
+```
+A (writes harvest_input for run R)  ──finishes──►
+B: python cre.py --run_noise_filter --run_id R --cache_file <db>
+   └─ exit 0 + JSON summary  ──►
+C (reads knowledge_queue)
+```
+The orchestrator **serialises** the steps: call B only after A has finished writing run R; call C only after B exits 0. B never polls or waits — sequencing is the orchestrator's job.
+
+## 7. Guarantees
+
+- **Recall-first:** only NOISE is dropped; KNOWLEDGE and UNCERTAIN always reach the queue (no security knowledge lost).
+- **Idempotent:** input rows are marked `processed`; re-invoking the same `run_id` is safe. `UNIQUE(content_hash)` collapses duplicate content.
+- **Error isolation:** an unparseable input row → marked `error` (not fatal); a failed LLM batch → those chunks become `UNCERTAIN` (never dropped); infrastructure failure → non-zero exit for the orchestrator to retry.
+
+---
+
+## Open enhancement (optional)
+
+Today the DB is passed via `--cache_file`. If you'd prefer 12-factor/env-based config, we can make `--run_noise_filter` fall back to `DATABASE_URL`/`DEV_DATABASE_URL` when `--cache_file` is omitted — say the word.