From 2d9fba46e9e0501d08adedc6cc4c0e828b7850f1 Mon Sep 17 00:00:00 2001 From: manshusainishab Date: Sun, 16 Aug 2026 04:51:40 +0530 Subject: [PATCH 1/3] docs(module-b): commit + sync A/C contracts and runbook --- .gitignore | 3 + docs/gsoc_2026_module_b/module_a_contract.md | 222 +++++++++++++++++++ docs/gsoc_2026_module_b/module_b_runbook.md | 121 ++++++++++ docs/gsoc_2026_module_b/module_c_contract.md | 152 +++++++++++++ 4 files changed, 498 insertions(+) create mode 100644 docs/gsoc_2026_module_b/module_a_contract.md create mode 100644 docs/gsoc_2026_module_b/module_b_runbook.md create mode 100644 docs/gsoc_2026_module_b/module_c_contract.md diff --git a/.gitignore b/.gitignore index 26b6ceb61..f6bad1914 100644 --- a/.gitignore +++ b/.gitignore @@ -69,6 +69,9 @@ standards_cache.sqlite !AGENTS.md !docs/faq.md !docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md +!docs/gsoc_2026_module_b/module_a_contract.md +!docs/gsoc_2026_module_b/module_c_contract.md +!docs/gsoc_2026_module_b/module_b_runbook.md ### Dev DBDumps *.sql diff --git a/docs/gsoc_2026_module_b/module_a_contract.md b/docs/gsoc_2026_module_b/module_a_contract.md new file mode 100644 index 000000000..c355b5e55 --- /dev/null +++ b/docs/gsoc_2026_module_b/module_a_contract.md @@ -0,0 +1,222 @@ +# Module A → Module B Input Contract + +**Audience:** the GSoC 2026 contributor implementing Module A (Information Harvesting — nightly cron job that fetches content from OWASP repos and feeds). **Status:** draft **v0.4** (2026-08-16). Reconciled against the orchestrated-pipeline hand-off (DB table, not a JSONL file) and Module C's shipped consumer (`feed_item` rss locator kind). Pending review by mentors and the Module A contributor. + +This document specifies the format Module A emits so that Module B (Noise/Relevance Filter) can consume it. Module B is the only downstream consumer in v1; any change to this format is a breaking change for B. + +--- + +## Changelog (v0.3 → v0.4) + +- **Delivery moved from a JSONL file to a DB table.** The orchestrated pipeline (see `orchestrator_integration_design.md`) supersedes the earlier "JSONL file via `cre.py --filter_changes `" transport. Module A now writes each record as a row in the shared Postgres `harvest_input` table (JSONB `payload` + top-level `pipeline_run_id` + `status`); Module B reads the pending rows for a run. The `--filter_changes` CLI was never built; Module B's entry point is `cre.py --run_noise_filter --run_id `. +- **rss `locator.kind` reserved value is `feed_item`** (was `feed_post`), aligning with Module C's shipped consumer (PR #1011), which addresses rss rows by `locator_kind = "feed_item"`. github rows are unchanged (`repo_path`). rss is still not emitted; this only fixes the reserved name so all three modules agree. + +## Changelog (v0.2 → v0.3) + +Driven by reconciliation with Module A's actual mock output. The shape is significantly different from what the v0.2 draft anticipated. + +- **Structural shift:** record fields are no longer flat. Three nested objects now exist: `source` (provenance), `span` (chunk position within parent artifact), `locator` (addressable identity). +- **New top-level fields:** `schema_version`, `chunk_id`, `pipeline_run_id`. +- **Renames / relocations:** + - `chunk_text` → `text` + - `source_type` → `source.type` + - `repo` → `source.repo` + - `commit_sha` → `source.commit_sha` + - `author_date` → `source.committed_at` + - `file_path` → `locator.path` (also `locator.id` mirrors for now) + - `chunk_index` → `span.index` +- **New `span` payload:** beyond `index` (which `chunk_index` already provided), `span` also carries `total`, `heading_path` (the markdown heading breadcrumb), `start_char_idx`/`end_char_idx`, `start_line`/`end_line`. +- **`content_hash` removed.** Module A does not emit a content hash. **Module B computes its own** by applying the v0.2 normalization rules (NFC, line endings, whitespace, code-fence preservation) and SHA-256-ing the result. Used by B as the `knowledge_queue` dedup key. +- **`commit_message` removed.** Module A does not emit commit messages. Module B's LLM prompt now uses `span.heading_path` as the semantic context signal instead (e.g. `["Authentication", "JWT"]` is a richer disambiguator than a commit message). +- **`source.type = "rss"` is reserved.** Mock data is github-only; the discriminated-union schema accepts RSS shape so we're ready when Module A emits feed records. +- **Mock note:** the mock data uses placeholder values like `"abc123"` for `commit_sha` and `"…"` (encoded as corrupted UTF-8 `â¦` in the source) for path segments inside `chunk_id`. Production must emit real 40-char SHAs and clean UTF-8. + +--- + +## Transport + +- **Format:** one JSON object per chunk — the record described below — stored verbatim as the JSONB `payload` of a `harvest_input` row. (Module B's local test fixtures still keep the same records as JSONL; the on-the-wire shape of a single record is identical either way.) +- **Delivery:** Module A writes each record as a row in the shared Postgres **`harvest_input`** table — columns `payload` (JSONB, the record), `pipeline_run_id` (top-level, run-scoping), `status` (`pending` → set by A; `processed`/`error` → set by B), plus `id`/`created_at`. Module B reads that run's pending rows (`cre.py --run_noise_filter --run_id `), classifies, and writes keepers to `knowledge_queue`. See `orchestrator_integration_design.md` and `module_b_runbook.md`. *(The earlier "JSONL file via `cre.py --filter_changes`" delivery is superseded and was never implemented.)* +- **Future (out of scope for v1):** object-storage URL (S3/MinIO) for large or out-of-band payloads. +- **Record size:** governed by Module A's chunking config (default `max_chars=4000`). Module B truncates internally at 1500 chars before sending to the LLM. + +## Required fields (top level) + +| Field | Type | Constraints | +|---|---|---| +| `schema_version` | string | E.g. `"0.2.0"`. Pinned by Module A; B reads but does not validate version semantics. | +| `chunk_id` | string | Module-A-stable identifier. Format observed in mock: `chk::` (i.e. `chk:art:OWASP/::`). | +| `artifact_id` | string | Identifier for the parent document. Format observed in mock: `art::`. Stable across chunks of the same artifact. | +| `pipeline_run_id` | string | Identifier for the Module A pipeline execution that produced this record. E.g. `"20260201T020000Z"`. Used by B for audit / replay grouping. | +| `text` | string | The normalized chunk content. Markdown markers (`#`, `**`, code fences) preserved; HTML stripped; whitespace collapsed in prose but preserved inside code fences (`` ``` ``…`` ``` ``) and `
` blocks. |
+| `span` | object | Position metadata. See below. |
+| `source` | object | Provenance discriminator (`source.type`). See below. |
+| `locator` | object | Addressable identity. See below. |
+
+## Required `span` payload
+
+| Field | Type | Constraints |
+|---|---|---|
+| `index` | int ≥ 0 | Zero-based chunk index within the parent artifact. |
+| `total` | int ≥ 1 | Total chunks Module A produced from this artifact. |
+| `heading_path` | array of strings | Breadcrumb of enclosing markdown headings (e.g. `["Authentication", "JWT"]`). Empty array if the chunk precedes all headings. Used as a semantic signal in B's LLM prompt. |
+| `start_char_idx` | int ≥ 0 (optional) | Character index of chunk start in the normalized artifact text. |
+| `end_char_idx` | int ≥ 0 (optional) | Character index of chunk end (exclusive). |
+| `start_line` | int ≥ 0 (optional) | 1-based line number of chunk start in the normalized artifact. |
+| `end_line` | int ≥ 0 (optional) | 1-based line number of chunk end. |
+
+## Required `source` payload — discriminated union
+
+`source.type` discriminates between provenance shapes.
+
+**When `source.type = "github"`:**
+
+| Field | Type | Constraints |
+|---|---|---|
+| `type` | string | Literal `"github"`. |
+| `repo` | string | Format `^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$`. Allows OWASP dots/dashes. |
+| `commit_sha` | string | Production: 40-char hex SHA-1. Mock: 6+ char placeholders accepted. |
+| `committed_at` | string | ISO-8601 timestamp. |
+
+**When `source.type = "rss"` (reserved, not yet emitted):**
+
+| Field | Type | Constraints |
+|---|---|---|
+| `type` | string | Literal `"rss"`. |
+| `feed_url` | string | The canonical feed URL Module A subscribed to. |
+| `post_guid` | string | The `` or `` of the post within the feed. |
+| `post_published_at` | string (optional) | ISO-8601 timestamp. |
+
+## Required `locator` payload
+
+| Field | Type | Constraints |
+|---|---|---|
+| `kind` | string | Scheme. `"repo_path"` for github sources today. Reserved: `"feed_item"` for RSS (matches Module C's consumer, PR #1011). |
+| `id` | string | Unique identity within the scheme. For `repo_path`: the file path. |
+| `path` | string | Convenience duplicate of `id` for `repo_path` (`id == path`). |
+
+## Example record (github source — mock-shaped)
+
+```json
+{
+  "schema_version": "0.2.0",
+  "chunk_id": "chk:art:OWASP/ASVS:4.0/en/0x12-V3-Authentication.md:0",
+  "artifact_id": "art:OWASP/ASVS:4.0/en/0x12-V3-Authentication.md",
+  "pipeline_run_id": "20260201T020000Z",
+  "text": "Authentication should use MFA",
+  "span": {
+    "index": 0,
+    "total": 3,
+    "heading_path": ["Authentication", "JWT"],
+    "start_char_idx": 0,
+    "end_char_idx": 98,
+    "start_line": 10,
+    "end_line": 12
+  },
+  "source": {
+    "type": "github",
+    "repo": "OWASP/ASVS",
+    "commit_sha": "abc123",
+    "committed_at": "2026-02-01T01:00:00Z"
+  },
+  "locator": {
+    "kind": "repo_path",
+    "id": "4.0/en/0x12-V3-Authentication.md",
+    "path": "4.0/en/0x12-V3-Authentication.md"
+  }
+}
+```
+
+## Content hashing — Module B's responsibility
+
+Module A does **not** emit `content_hash`. Module B computes one on ingest in `application/utils/noise_filter/hashing.py`:
+
+1. Apply v0.2 normalization rules to `text`:
+   - Unicode NFC
+   - CRLF / CR → LF
+   - Trailing whitespace per line stripped
+   - Leading/trailing blank lines stripped
+   - Runs of spaces/tabs in prose collapsed to one space
+   - Whitespace inside fenced code blocks (`` ``` `` … `` ``` ``) and `
` blocks preserved verbatim
+2. `content_hash = hashlib.sha256(normalized_text.encode("utf-8")).hexdigest()`
+
+The hash becomes the `UniqueConstraint` key on `knowledge_queue` — re-feeding identical normalized content (e.g. mirrored docs, replayed pipeline runs) collapses to one queue row.
+
+**Future:** if Module A starts emitting `content_hash`, B switches via `CRE_NOISE_FILTER_TRUST_A_HASH=true` to skip recomputation. Until then, B is self-sufficient.
+
+## Normalization ownership
+
+- **Module A owns "normalize for chunking":** semantic chunking, heading-path tracking, char/line offsets.
+- **Module B owns "normalize for hashing" + "defensive sanitization":** the rules above (for `content_hash`), plus TRACT-style sanitize.py (zero-width chars, PDF ligatures, hyphenation rejoin) as Stage 1.5 of B's pipeline.
+
+If Module A's own normalization differs from B's (e.g. A doesn't collapse prose whitespace), B's hash still works — B always normalizes before hashing. The two normalizations don't need to agree.
+
+## Path-filtering ownership
+
+- **Module A owns coarse exclusion:** the `paths_exclude` globs in its source config (e.g. `["**/package-lock.json", "**/CNAME"]`). These never reach B.
+- **Module B owns fine-grained noise filtering:** `application/utils/noise_filter/noise_patterns.yaml`. Catches things A's source config didn't anticipate.
+
+If both modules block the same path, that's fine — B silently no-ops. The two lists drift independently.
+
+## Stability guarantees
+
+- **Module B reads only the fields listed above** (and ignores everything else). Extra fields Module A adds — `pr_number`, `author`, `tags`, `supersedes_artifact_id`, etc. — are silently accepted (`extra="ignore"` on B's Pydantic models). Safe to extend.
+- **Renaming or removing any required field is breaking.** Requires a contract version bump and coordinated changes in `application/utils/noise_filter/schemas.py`.
+- **Changing the semantics of a field is breaking** even if the name stays the same. Example: switching `locator.path` to absolute paths would break B's regex pre-filter.
+
+## Idempotency on Module B's side
+
+Module B's `knowledge_queue` uses `UniqueConstraint(content_hash)` as the dedup key. Consequences:
+- Re-feeding the same normalized content is a no-op.
+- The same content reaching B via two different sources collapses to one row.
+- To force re-classification (e.g. prompt changed), Module B will provide an `--allow_duplicate_hash` flag on the CLI. Out of scope for v1.
+
+## Versioning
+
+This contract is **v0.3** (draft). When ratified, becomes v1.0. semver applies:
+- v1.X = additive, non-breaking field additions.
+- v2.0 = breaking changes.
+
+The version applies to *this contract*, not to Module A's release cadence.
+
+## JSON Schema artifact
+
+The machine-readable schema is generated from Module B's Pydantic models and committed at:
+
+```
+docs/gsoc_2026_module_b/module_a_contract.schema.json
+```
+
+Both modules SHOULD validate against it in CI. Module B's Pydantic model (`application/utils/noise_filter/schemas.ChangeRecord`) is the canonical source; the JSON Schema file is derived via `ChangeRecord.model_json_schema()`.
+
+## Test fixtures
+
+Module B keeps fixtures at:
+
+```
+application/tests/noise_filter/fixtures/
+├── module_a_mock.jsonl          # Module A's 20-record mock (canonical, what A actually emits)
+├── candidate_commits.json       # B's own stand-in harvest of ~100 OWASP commits in Module A's shape
+└── labeled_data.json            # candidate_commits.json + KNOWLEDGE/NOISE/UNCERTAIN labels
+```
+
+Module A contributors are welcome to add fixtures here as PRs — small files (≤30 records each) covering edge cases (PDF-extracted text, HTML-derived chunks, RSS posts) help Module B's regex and prompt iteration.
+
+## Out-of-scope for this contract
+
+- How Module A produces these records (GitHub API vs `git log` vs webhook vs feed poll).
+- Where Module A stores raw artifacts before chunking.
+- Failure handling on the Module A side (rate-limit retries, partial commits).
+- Authentication / API keys (Module A's concern).
+- Module A's source-config schema (`schema_version`, `sources`, `chunking`) — that's internal to A.
+
+## Open questions for the Module A contributor
+
+These don't gate this contract:
+
+1. **`content_hash` — will A start emitting it?** B can use A's hash if shipped; otherwise B computes. Either is fine.
+2. **What does `locator.kind` enumerate?** `"repo_path"` is the only value observed for github. For RSS, the reserved value is `"feed_item"` (Module C already consumes rss rows under that kind, PR #1011) — please confirm Module A will emit `"feed_item"` when feeds go live.
+3. **`pipeline_run_id` format** — is `YYYYMMDDTHHMMSSZ` stable, or might it become a UUID? B doesn't parse the value (just stores it as a string), but reviewers may want to know.
+4. **Will `chunk_id`'s embedded path component be URL-escaped** in production? In the mock it's the raw path (with `/` characters); that's fine for B but worth confirming.
+
+Open these on Slack `#project-opencre` or as PR comments on Module B's Week 1 PR.
diff --git a/docs/gsoc_2026_module_b/module_b_runbook.md b/docs/gsoc_2026_module_b/module_b_runbook.md
new file mode 100644
index 000000000..a39b1414c
--- /dev/null
+++ b/docs/gsoc_2026_module_b/module_b_runbook.md
@@ -0,0 +1,121 @@
+# Module B — How to Run It (Orchestrator Runbook)
+
+**Audience:** whoever builds/operates the daily orchestrator. **Status:** v0.2 (2026-08-16).
+Companion to `orchestrator_integration_design.md` (the *why*) and `module_c_contract.md` (B→C). This is the *how*.
+
+Module B is a **stateless batch step**: the orchestrator invokes it once per harvest run; it reads that run's chunks from a DB table, classifies them, writes the keepers to another table, and exits with a JSON summary. It does not run continuously and does not schedule itself.
+
+---
+
+## 1. One-time setup
+
+**Tables.** Module B owns two tables, created by its Alembic migration
+`d4e5f6a7b8c9_add_module_b_tables` (part of the chain — other modules' migrations
+now chain after it, so it is no longer the head):
+- `harvest_input` — Module A writes here; B reads.
+- `knowledge_queue` — B writes here; Module C reads.
+
+Apply with:
+```bash
+FLASK_APP=cre.py FLASK_CONFIG=development flask db upgrade
+```
+A full from-empty `flask db upgrade` on Postgres runs the whole chain, creating
+Module B's tables at migration `d4e5f6a7b8c9` (now mid-chain; the current head
+moves as other modules add migrations after it — the from-empty upgrade still
+reaches B's tables regardless). The earlier `uq_pair` duplicate-index bug is
+fixed and merged. One caveat: **C's pgvector migration (`c7d8e9f0a1b2`)
+requires `CRE_EMBED_EXPECTED_DIM` to be set** on an empty DB (it can't infer the
+vector dimension with no embeddings yet) — a pre-existing requirement of that
+migration, e.g. `CRE_EMBED_EXPECTED_DIM=3072 flask db upgrade`. Postgres needs
+the `vector` extension (use the `pgvector/pgvector` image or `CREATE EXTENSION
+vector;`).
+
+**Environment variables:**
+| Var | Purpose | Default |
+|---|---|---|
+| `GEMINI_API_KEY` | LLM credential (Gemini) | — (required) |
+| `CRE_NOISE_FILTER_LLM_MODEL` | classification model | `gemini/gemini-2.5-flash-lite` |
+| `CRE_NOISE_FILTER_BATCH_SIZE` | chunks per LLM request | `10` |
+| `CRE_NOISE_FILTER_MAX_CHARS` | per-chunk truncation | `1500` |
+| `CRE_NOISE_FILTER_CONFIDENCE_THRESHOLD` | KNOWLEDGE-below → UNCERTAIN | `0.8` |
+
+Module B needs no ML libraries (no torch/sentence-transformers) — just `litellm` (in the slim prod requirements) + the Gemini key.
+
+---
+
+## 2. The input table `harvest_input` (Module A writes)
+
+One row per harvested chunk:
+| Column | Notes |
+|---|---|
+| `id` | any stable PK |
+| `pipeline_run_id` | groups a run — B reads exactly this |
+| `status` | `pending` (A sets) → `processed`/`error` (B sets) |
+| `payload` | **JSONB** — Module A's ChangeRecord v0.3 (nested `source`/`span`/`locator`), written as-is |
+| `created_at` | timestamp |
+
+B reads `WHERE pipeline_run_id = :run_id AND status = 'pending'`.
+
+---
+
+## 3. Invoking Module B (the command the orchestrator runs)
+
+```bash
+python cre.py --run_noise_filter --run_id  --cache_file 
+```
+- `--run_id` — the run to process (**required**).
+- `--cache_file` — the database URL (e.g. `postgresql://user:pass@host:5432/opencre`).
+- `--noise_filter_dry_run` — optional; classify but write nothing / mark nothing (for testing).
+
+The process runs the gate (regex path filter → sanitize → LLM classify), writes keepers, marks the input rows, and exits.
+
+---
+
+## 4. Completion signal (how the orchestrator knows it's done)
+
+- **Exit code `0`** = success; **non-zero** = hard failure (e.g. DB unreachable) → safe to retry the same `run_id` (B is idempotent).
+- **stdout** = a one-line JSON summary:
+```json
+{"run_id":"20260201T020000Z","read":512,"parse_errors":0,"dropped_noise":172,
+ "kept_knowledge":300,"kept_uncertain":40,"inserted":338,"deduped":2,
+ "dry_run":false,"status":"ok"}
+```
+| Field | Meaning |
+|---|---|
+| `read` | input rows for the run |
+| `parse_errors` | payloads that failed validation (rows marked `error`) |
+| `dropped_noise` | dropped as NOISE (regex + LLM) — not queued |
+| `kept_knowledge` / `kept_uncertain` | classified as KNOWLEDGE / UNCERTAIN |
+| `inserted` | rows written to `knowledge_queue` |
+| `deduped` | keepers skipped as duplicate content |
+| `status` | `ok` (per-chunk errors are contained, not fatal) |
+
+---
+
+## 5. The output table `knowledge_queue` (Module C reads)
+
+B inserts `KNOWLEDGE` and `UNCERTAIN` rows (never `NOISE`), deduped on `content_hash`. Module C reads unconsumed rows and sets `consumed_at`. Full schema + read query: `module_c_contract.md` (v0.2).
+
+---
+
+## 6. Orchestrator sequencing
+
+```
+A (writes harvest_input for run R)  ──finishes──►
+B: python cre.py --run_noise_filter --run_id R --cache_file 
+   └─ exit 0 + JSON summary  ──►
+C (reads knowledge_queue)
+```
+The orchestrator **serialises** the steps: call B only after A has finished writing run R; call C only after B exits 0. B never polls or waits — sequencing is the orchestrator's job.
+
+## 7. Guarantees
+
+- **Recall-first:** only NOISE is dropped; KNOWLEDGE and UNCERTAIN always reach the queue (no security knowledge lost).
+- **Idempotent:** input rows are marked `processed`; re-invoking the same `run_id` is safe. `UNIQUE(content_hash)` collapses duplicate content.
+- **Error isolation:** an unparseable input row → marked `error` (not fatal); a failed LLM batch → those chunks become `UNCERTAIN` (never dropped); infrastructure failure → non-zero exit for the orchestrator to retry.
+
+---
+
+## Open enhancement (optional)
+
+Today the DB is passed via `--cache_file`. If you'd prefer 12-factor/env-based config, we can make `--run_noise_filter` fall back to `DATABASE_URL`/`DEV_DATABASE_URL` when `--cache_file` is omitted — say the word.
diff --git a/docs/gsoc_2026_module_b/module_c_contract.md b/docs/gsoc_2026_module_b/module_c_contract.md
new file mode 100644
index 000000000..de2036158
--- /dev/null
+++ b/docs/gsoc_2026_module_b/module_c_contract.md
@@ -0,0 +1,152 @@
+# Module B → Module C Output Contract
+
+**Audience:** the GSoC 2026 contributor implementing Module C (The Librarian — vector + cross-encoder mapping of filtered knowledge chunks to existing CRE nodes). **Status:** draft **v0.3** (2026-08-16). Reconciled with the orchestrated-pipeline design (`orchestrator_integration_design.md`) and with Module C's shipped consumer (PR #1011), which now mirrors this table column-for-column.
+
+This document specifies how Module C reads Module B's output. Module B produces; Module C consumes; Module D's HITL UI may also read for review.
+
+---
+
+## Changelog (v0.2 → v0.3)
+
+- **Verified against Module C's shipped consumer (PR #1011).** C's `KnowledgeQueueItem` now mirrors this table column-for-column (all 23 columns, both `github` and `rss` provenance branches). C reads with `consumed_at IS NULL AND llm_label = 'KNOWLEDGE'` and stamps `consumed_at` as its only write — matching this contract. UNCERTAIN rows are left for Module D.
+- **`source_committed_at` type corrected to `String`.** Module B stores it as an ISO-8601 **string** (the value Module A emits, unparsed), not a `DateTime`. Module C parses it to a datetime on read (Pydantic coercion) — no conflict, but the storage type in this contract now matches Module B's actual model/migration.
+
+## Changelog (v0.1 → v0.2)
+
+- **Orchestrated pipeline:** B is now invoked by the daily **orchestrator** (not a manual CLI run). B reads Module A's chunks from a Postgres table, classifies, writes keepers to `knowledge_queue`, and signals the orchestrator "done"; the orchestrator then calls C. See `orchestrator_integration_design.md`.
+- **Schema aligned to Module A contract v0.3:** provenance columns now mirror A's nested `source` / `span` / `locator` record (confirmed unchanged by A on 2026-07-16), instead of the old flat `source_repo` / `source_path` / `source_commit_sha`.
+- **Dedup key changed:** `UNIQUE(content_hash)` (B-computed on the normalized text) replaces `UNIQUE(source_commit_sha, source_path)`. The same normalized content reaching B via two sources or two runs collapses to one row.
+- **Source-type-aware:** schema and read query now support the `source.type` discriminator (`github` today; `rss` reserved).
+
+---
+
+## Transport
+
+- **Format:** SQL table `knowledge_queue` in the shared OpenCRE Postgres database (the same DB the orchestrator uses to hand off between modules).
+- **Backend:** SQLite for dev/CI/harness; **PostgreSQL** in production (Module C uses pgvector).
+- **Producer:** Module B, when the orchestrator invokes it for a run. B inserts `KNOWLEDGE` and `UNCERTAIN` rows; `NOISE` is dropped (recorded only in B's run summary/audit).
+- **SQLAlchemy model:** `KnowledgeQueueItem` in `application/database/db.py` (added by Module B at integration, with an Alembic migration that must pass `make alembic-guardrail`).
+
+Why a table (not a Redis queue, JSONL handoff, or upstream's `StagedChangeSet`): durability + history + `consumed_at` semantics + dedup, and it fits the orchestrator's DB-handoff model. Upstream's `ImportRun`/`StandardSnapshot`/`StagedChangeSet` solve *standards versioning*, not filtered-chunk consumption.
+
+## Table schema
+
+```python
+class KnowledgeQueueItem(BaseModel):
+    __tablename__ = "knowledge_queue"
+    id             = sqla.Column(sqla.String, primary_key=True, default=generate_uuid)
+    content_hash   = sqla.Column(sqla.String(64), nullable=False)   # B-computed dedup key
+    # provenance / traceability (from Module A's v0.3 record)
+    chunk_id        = sqla.Column(sqla.String, nullable=False)
+    artifact_id     = sqla.Column(sqla.String, nullable=False)
+    pipeline_run_id = sqla.Column(sqla.String, nullable=False)
+    schema_version  = sqla.Column(sqla.String, nullable=False)
+    source_type     = sqla.Column(sqla.String, nullable=False)       # "github" | "rss"
+    source_repo         = sqla.Column(sqla.String, nullable=True)     # github-only
+    source_commit_sha   = sqla.Column(sqla.String, nullable=True)
+    source_committed_at = sqla.Column(sqla.String, nullable=True)     # ISO-8601 string (C parses)
+    feed_url        = sqla.Column(sqla.String, nullable=True)         # rss-only
+    post_guid       = sqla.Column(sqla.String, nullable=True)
+    locator_kind    = sqla.Column(sqla.String, nullable=False)
+    locator_path    = sqla.Column(sqla.String, nullable=False)
+    span_index      = sqla.Column(sqla.Integer, nullable=False)
+    span_total      = sqla.Column(sqla.Integer, nullable=False)
+    span_heading_path = sqla.Column(sqla.Text, nullable=True)         # JSON-encoded list[str]
+    # payload + B's verdict
+    text          = sqla.Column(sqla.Text, nullable=False)
+    llm_label     = sqla.Column(sqla.String, nullable=False)          # KNOWLEDGE | UNCERTAIN
+    confidence    = sqla.Column(sqla.Float, nullable=False)
+    llm_reasoning = sqla.Column(sqla.Text, nullable=True)
+    created_at    = sqla.Column(sqla.DateTime, nullable=False, server_default=sqla.func.now())
+    consumed_at   = sqla.Column(sqla.DateTime, nullable=True)
+    __table_args__ = (
+        sqla.Index("ix_knowledge_queue_unconsumed", "consumed_at"),
+        sqla.UniqueConstraint("content_hash", name="uq_content_hash"),
+    )
+```
+
+| Column | Purpose for Module C |
+|---|---|
+| `id` | Stable UUID. Use as the source identity for CRE-mapping logs. |
+| `content_hash` | B's SHA-256 of the normalized **original** chunk text; the dedup key (stable across sanitizer changes). |
+| `chunk_id`, `artifact_id`, `pipeline_run_id`, `schema_version` | Traceability back to Module A's record (and to Module D's review UI). |
+| `source_type` + `source_*` / `feed_*` | Provenance discriminator. `github` populates `source_repo`/`source_commit_sha`/`source_committed_at`; `rss` populates `feed_url`/`post_guid`. |
+| `locator_kind`, `locator_path` | Addressable location (`repo_path` → file path today). |
+| `span_index`, `span_total`, `span_heading_path` | Chunk position + heading breadcrumb (context for review/mapping). |
+| `text` | The chunk text as harvested by Module A (canonical), for C to map to a CRE node. B's defensive sanitization is applied only to the LLM's classification input, so the stored `text` and `content_hash` stay canonical/stable. |
+| `llm_label` | `KNOWLEDGE` or `UNCERTAIN` (B never writes `NOISE` here). |
+| `confidence` | 0.0–1.0, B's LLM confidence. |
+| `llm_reasoning` | Optional one-line rationale (debugging / Module D UI). |
+| `created_at` | When B wrote the row. FIFO ordering. |
+| `consumed_at` | NULL = pending. Module C sets `NOW()` after successful ingest. |
+
+## Canonical read query (source-type-aware)
+
+```sql
+SELECT
+    id,
+    CASE source_type
+      WHEN 'github' THEN source_repo || '@' || source_commit_sha || ':' || locator_path
+      WHEN 'rss'    THEN feed_url || '#' || post_guid
+    END AS source,
+    text,
+    confidence
+FROM knowledge_queue
+WHERE consumed_at IS NULL
+  AND llm_label = 'KNOWLEDGE'   -- optional: skip UNCERTAIN (those are for Module D HITL)
+ORDER BY created_at
+LIMIT :batch_size;
+```
+
+The synthesized `source` gives the frozen `{source, text, confidence}` payload C originally requested:
+
+```json
+{"source": "OWASP/ASVS@abc123:4.0/en/0x12-V3-Authentication.md", "text": "...", "confidence": 0.91}
+```
+
+## Consumption semantics
+
+After mapping a batch, Module C MUST mark rows consumed:
+
+```sql
+UPDATE knowledge_queue SET consumed_at = NOW() WHERE id IN (:ids);
+```
+
+- **Idempotency on retries:** un-marked rows keep `consumed_at IS NULL` and are picked up next poll. `UNIQUE(content_hash)` prevents B from inserting the same logical row twice.
+- **Ordering:** FIFO by `created_at`; use `id` as a tiebreaker for identical timestamps.
+- **Concurrent consumers:** `consumed_at IS NULL` + row-level `UPDATE ... WHERE consumed_at IS NULL` is safe; for multi-consumer, use `SELECT ... FOR UPDATE SKIP LOCKED`.
+
+## UNCERTAIN row policy
+
+- B writes `llm_label = 'UNCERTAIN'` when the LLM returned UNCERTAIN, or when the response failed to parse / the batch failed (confidence 0.0).
+- **Recommended C behavior:** filter to `KNOWLEDGE` for normal mapping; UNCERTAIN rows are for Module D's HITL review.
+- **If Module D isn't live yet:** C MAY treat UNCERTAIN as KNOWLEDGE (slightly higher false-positive rate); document the choice.
+
+## Stability guarantees
+
+- **C reads only the columns in the canonical query.** B may add columns; existing ones won't be renamed/removed without a version bump.
+- **The `{source, text, confidence}` projection is frozen for v1.0**, even if B's internal columns evolve.
+- **`consumed_at` is never reset** to NULL on an already-consumed row.
+
+## Versioning
+
+This contract is **v0.2 (draft)**. Becomes v1.0 when both contributors agree. semver applies as in the Module A contract.
+
+## Test fixtures and integration
+
+- Module B will produce a real-data fixture (a SQL dump of `knowledge_queue` from a pipeline run) so Module C can replay consumption end-to-end without A or B running live.
+- First end-to-end live test: B fills the table on the shared dev Postgres + pgvector; C polls and maps a small batch. Coordinate in Slack `#project-opencre`.
+
+## Out-of-scope for this contract
+
+- How C maps chunks to CRE nodes (vector / cross-encoder / hybrid) — C's design space.
+- Module D's HITL UI integration — separate contract (`module_d_contract.md`, future).
+- GC of fully-consumed rows — neither B nor C deletes in v1.
+- Read replicas / sharding — single-master Postgres for v1.
+
+## Open questions for the Module C contributor
+
+1. **`cre_mapping_id` column?** Should C populate a back-link to the resulting CRE/Standard/Link row after mapping? Useful for round-trip audit, but couples B/C schemas.
+2. **`last_seen_at` telemetry?** Or is `consumed_at` enough?
+3. **`text` column type** on Postgres for large chunks — `Text` fine, or compressed/`bytea`?
+4. **Polling vs LISTEN/NOTIFY?** v1 assumes polling; push only if latency becomes a problem.

From 5687544dd41f1ddb9b3fa957fc14b234ec1d7e34 Mon Sep 17 00:00:00 2001
From: manshusainishab 
Date: Sun, 16 Aug 2026 05:12:30 +0530
Subject: [PATCH 2/3] docs(module-b): drop open-questions + local design-doc
 refs from contracts

---
 docs/gsoc_2026_module_b/module_a_contract.md | 15 ++-------------
 docs/gsoc_2026_module_b/module_b_runbook.md  |  2 +-
 docs/gsoc_2026_module_b/module_c_contract.md | 13 +++----------
 3 files changed, 6 insertions(+), 24 deletions(-)

diff --git a/docs/gsoc_2026_module_b/module_a_contract.md b/docs/gsoc_2026_module_b/module_a_contract.md
index c355b5e55..359d51893 100644
--- a/docs/gsoc_2026_module_b/module_a_contract.md
+++ b/docs/gsoc_2026_module_b/module_a_contract.md
@@ -8,7 +8,7 @@ This document specifies the format Module A emits so that Module B (Noise/Releva
 
 ## Changelog (v0.3 → v0.4)
 
-- **Delivery moved from a JSONL file to a DB table.** The orchestrated pipeline (see `orchestrator_integration_design.md`) supersedes the earlier "JSONL file via `cre.py --filter_changes `" transport. Module A now writes each record as a row in the shared Postgres `harvest_input` table (JSONB `payload` + top-level `pipeline_run_id` + `status`); Module B reads the pending rows for a run. The `--filter_changes` CLI was never built; Module B's entry point is `cre.py --run_noise_filter --run_id `.
+- **Delivery moved from a JSONL file to a DB table.** The orchestrated pipeline supersedes the earlier "JSONL file via `cre.py --filter_changes `" transport. Module A now writes each record as a row in the shared Postgres `harvest_input` table (JSONB `payload` + top-level `pipeline_run_id` + `status`); Module B reads the pending rows for a run. The `--filter_changes` CLI was never built; Module B's entry point is `cre.py --run_noise_filter --run_id `.
 - **rss `locator.kind` reserved value is `feed_item`** (was `feed_post`), aligning with Module C's shipped consumer (PR #1011), which addresses rss rows by `locator_kind = "feed_item"`. github rows are unchanged (`repo_path`). rss is still not emitted; this only fixes the reserved name so all three modules agree.
 
 ## Changelog (v0.2 → v0.3)
@@ -36,7 +36,7 @@ Driven by reconciliation with Module A's actual mock output. The shape is signif
 ## Transport
 
 - **Format:** one JSON object per chunk — the record described below — stored verbatim as the JSONB `payload` of a `harvest_input` row. (Module B's local test fixtures still keep the same records as JSONL; the on-the-wire shape of a single record is identical either way.)
-- **Delivery:** Module A writes each record as a row in the shared Postgres **`harvest_input`** table — columns `payload` (JSONB, the record), `pipeline_run_id` (top-level, run-scoping), `status` (`pending` → set by A; `processed`/`error` → set by B), plus `id`/`created_at`. Module B reads that run's pending rows (`cre.py --run_noise_filter --run_id `), classifies, and writes keepers to `knowledge_queue`. See `orchestrator_integration_design.md` and `module_b_runbook.md`. *(The earlier "JSONL file via `cre.py --filter_changes`" delivery is superseded and was never implemented.)*
+- **Delivery:** Module A writes each record as a row in the shared Postgres **`harvest_input`** table — columns `payload` (JSONB, the record), `pipeline_run_id` (top-level, run-scoping), `status` (`pending` → set by A; `processed`/`error` → set by B), plus `id`/`created_at`. Module B reads that run's pending rows (`cre.py --run_noise_filter --run_id `), classifies, and writes keepers to `knowledge_queue`. See `module_b_runbook.md` for the operational how-to. *(The earlier "JSONL file via `cre.py --filter_changes`" delivery is superseded and was never implemented.)*
 - **Future (out of scope for v1):** object-storage URL (S3/MinIO) for large or out-of-band payloads.
 - **Record size:** governed by Module A's chunking config (default `max_chars=4000`). Module B truncates internally at 1500 chars before sending to the LLM.
 
@@ -209,14 +209,3 @@ Module A contributors are welcome to add fixtures here as PRs — small files (
 - Failure handling on the Module A side (rate-limit retries, partial commits).
 - Authentication / API keys (Module A's concern).
 - Module A's source-config schema (`schema_version`, `sources`, `chunking`) — that's internal to A.
-
-## Open questions for the Module A contributor
-
-These don't gate this contract:
-
-1. **`content_hash` — will A start emitting it?** B can use A's hash if shipped; otherwise B computes. Either is fine.
-2. **What does `locator.kind` enumerate?** `"repo_path"` is the only value observed for github. For RSS, the reserved value is `"feed_item"` (Module C already consumes rss rows under that kind, PR #1011) — please confirm Module A will emit `"feed_item"` when feeds go live.
-3. **`pipeline_run_id` format** — is `YYYYMMDDTHHMMSSZ` stable, or might it become a UUID? B doesn't parse the value (just stores it as a string), but reviewers may want to know.
-4. **Will `chunk_id`'s embedded path component be URL-escaped** in production? In the mock it's the raw path (with `/` characters); that's fine for B but worth confirming.
-
-Open these on Slack `#project-opencre` or as PR comments on Module B's Week 1 PR.
diff --git a/docs/gsoc_2026_module_b/module_b_runbook.md b/docs/gsoc_2026_module_b/module_b_runbook.md
index a39b1414c..04ec5b30e 100644
--- a/docs/gsoc_2026_module_b/module_b_runbook.md
+++ b/docs/gsoc_2026_module_b/module_b_runbook.md
@@ -1,7 +1,7 @@
 # Module B — How to Run It (Orchestrator Runbook)
 
 **Audience:** whoever builds/operates the daily orchestrator. **Status:** v0.2 (2026-08-16).
-Companion to `orchestrator_integration_design.md` (the *why*) and `module_c_contract.md` (B→C). This is the *how*.
+Companion to `module_c_contract.md` (the B→C contract). This is the operational *how*.
 
 Module B is a **stateless batch step**: the orchestrator invokes it once per harvest run; it reads that run's chunks from a DB table, classifies them, writes the keepers to another table, and exits with a JSON summary. It does not run continuously and does not schedule itself.
 
diff --git a/docs/gsoc_2026_module_b/module_c_contract.md b/docs/gsoc_2026_module_b/module_c_contract.md
index de2036158..4068cadd9 100644
--- a/docs/gsoc_2026_module_b/module_c_contract.md
+++ b/docs/gsoc_2026_module_b/module_c_contract.md
@@ -1,6 +1,6 @@
 # Module B → Module C Output Contract
 
-**Audience:** the GSoC 2026 contributor implementing Module C (The Librarian — vector + cross-encoder mapping of filtered knowledge chunks to existing CRE nodes). **Status:** draft **v0.3** (2026-08-16). Reconciled with the orchestrated-pipeline design (`orchestrator_integration_design.md`) and with Module C's shipped consumer (PR #1011), which now mirrors this table column-for-column.
+**Audience:** the GSoC 2026 contributor implementing Module C (The Librarian — vector + cross-encoder mapping of filtered knowledge chunks to existing CRE nodes). **Status:** draft **v0.3** (2026-08-16). Reconciled with the orchestrated-pipeline hand-off and with Module C's shipped consumer (PR #1011), which now mirrors this table column-for-column.
 
 This document specifies how Module C reads Module B's output. Module B produces; Module C consumes; Module D's HITL UI may also read for review.
 
@@ -13,7 +13,7 @@ This document specifies how Module C reads Module B's output. Module B produces;
 
 ## Changelog (v0.1 → v0.2)
 
-- **Orchestrated pipeline:** B is now invoked by the daily **orchestrator** (not a manual CLI run). B reads Module A's chunks from a Postgres table, classifies, writes keepers to `knowledge_queue`, and signals the orchestrator "done"; the orchestrator then calls C. See `orchestrator_integration_design.md`.
+- **Orchestrated pipeline:** B is now invoked by the daily **orchestrator** (not a manual CLI run). B reads Module A's chunks from a Postgres table, classifies, writes keepers to `knowledge_queue`, and signals the orchestrator "done"; the orchestrator then calls C.
 - **Schema aligned to Module A contract v0.3:** provenance columns now mirror A's nested `source` / `span` / `locator` record (confirmed unchanged by A on 2026-07-16), instead of the old flat `source_repo` / `source_path` / `source_commit_sha`.
 - **Dedup key changed:** `UNIQUE(content_hash)` (B-computed on the normalized text) replaces `UNIQUE(source_commit_sha, source_path)`. The same normalized content reaching B via two sources or two runs collapses to one row.
 - **Source-type-aware:** schema and read query now support the `source.type` discriminator (`github` today; `rss` reserved).
@@ -135,7 +135,7 @@ This contract is **v0.2 (draft)**. Becomes v1.0 when both contributors agree. se
 ## Test fixtures and integration
 
 - Module B will produce a real-data fixture (a SQL dump of `knowledge_queue` from a pipeline run) so Module C can replay consumption end-to-end without A or B running live.
-- First end-to-end live test: B fills the table on the shared dev Postgres + pgvector; C polls and maps a small batch. Coordinate in Slack `#project-opencre`.
+- First end-to-end live test: B fills the table on the shared dev Postgres + pgvector; C polls and maps a small batch.
 
 ## Out-of-scope for this contract
 
@@ -143,10 +143,3 @@ This contract is **v0.2 (draft)**. Becomes v1.0 when both contributors agree. se
 - Module D's HITL UI integration — separate contract (`module_d_contract.md`, future).
 - GC of fully-consumed rows — neither B nor C deletes in v1.
 - Read replicas / sharding — single-master Postgres for v1.
-
-## Open questions for the Module C contributor
-
-1. **`cre_mapping_id` column?** Should C populate a back-link to the resulting CRE/Standard/Link row after mapping? Useful for round-trip audit, but couples B/C schemas.
-2. **`last_seen_at` telemetry?** Or is `consumed_at` enough?
-3. **`text` column type** on Postgres for large chunks — `Text` fine, or compressed/`bytea`?
-4. **Polling vs LISTEN/NOTIFY?** v1 assumes polling; push only if latency becomes a problem.

From c328158690d8cbd309eeea7b2cd67ae17361347e Mon Sep 17 00:00:00 2001
From: manshusainishab 
Date: Sun, 16 Aug 2026 05:56:05 +0530
Subject: [PATCH 3/3] docs(module-b): C consumes both KNOWLEDGE and UNCERTAIN
 rows

---
 docs/gsoc_2026_module_b/module_c_contract.md | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/docs/gsoc_2026_module_b/module_c_contract.md b/docs/gsoc_2026_module_b/module_c_contract.md
index 4068cadd9..1cdf178bb 100644
--- a/docs/gsoc_2026_module_b/module_c_contract.md
+++ b/docs/gsoc_2026_module_b/module_c_contract.md
@@ -8,7 +8,8 @@ This document specifies how Module C reads Module B's output. Module B produces;
 
 ## Changelog (v0.2 → v0.3)
 
-- **Verified against Module C's shipped consumer (PR #1011).** C's `KnowledgeQueueItem` now mirrors this table column-for-column (all 23 columns, both `github` and `rss` provenance branches). C reads with `consumed_at IS NULL AND llm_label = 'KNOWLEDGE'` and stamps `consumed_at` as its only write — matching this contract. UNCERTAIN rows are left for Module D.
+- **Module C consumes both `KNOWLEDGE` and `UNCERTAIN`.** The canonical read query now selects `llm_label IN ('KNOWLEDGE', 'UNCERTAIN')` (was `= 'KNOWLEDGE'`). B's `llm_label` is a confidence signal, not a routing directive: C consumes every non-NOISE row and decides internally which chunks need Module D's HITL review. This keeps recall-first intact end to end — no security chunk is stranded in the queue for a label that has no downstream consumer.
+- **Verified against Module C's shipped consumer (PR #1011).** C's `KnowledgeQueueItem` now mirrors this table column-for-column (all 23 columns, both `github` and `rss` provenance branches), and stamps `consumed_at` as its only write. (#1011 as shipped filters `llm_label = 'KNOWLEDGE'`; per the bullet above, its C0 read filter should be updated to `IN ('KNOWLEDGE', 'UNCERTAIN')`.)
 - **`source_committed_at` type corrected to `String`.** Module B stores it as an ISO-8601 **string** (the value Module A emits, unparsed), not a `DateTime`. Module C parses it to a datetime on read (Pydantic coercion) — no conflict, but the storage type in this contract now matches Module B's actual model/migration.
 
 ## Changelog (v0.1 → v0.2)
@@ -93,7 +94,7 @@ SELECT
     confidence
 FROM knowledge_queue
 WHERE consumed_at IS NULL
-  AND llm_label = 'KNOWLEDGE'   -- optional: skip UNCERTAIN (those are for Module D HITL)
+  AND llm_label IN ('KNOWLEDGE', 'UNCERTAIN')   -- C consumes both; escalation to Module D (HITL) is C's own decision, not B's label
 ORDER BY created_at
 LIMIT :batch_size;
 ```
@@ -118,9 +119,9 @@ UPDATE knowledge_queue SET consumed_at = NOW() WHERE id IN (:ids);
 
 ## UNCERTAIN row policy
 
-- B writes `llm_label = 'UNCERTAIN'` when the LLM returned UNCERTAIN, or when the response failed to parse / the batch failed (confidence 0.0).
-- **Recommended C behavior:** filter to `KNOWLEDGE` for normal mapping; UNCERTAIN rows are for Module D's HITL review.
-- **If Module D isn't live yet:** C MAY treat UNCERTAIN as KNOWLEDGE (slightly higher false-positive rate); document the choice.
+- B writes `llm_label = 'UNCERTAIN'` when the LLM returned UNCERTAIN (a genuine borderline chunk), or when classification failed — the batch errored or the response didn't parse (`confidence = 0.0`). B never drops these: recall-first means an UNCERTAIN chunk may still carry security signal.
+- **`llm_label` is B's confidence signal, not a routing directive.** Module C consumes **both** `KNOWLEDGE` and `UNCERTAIN` (`llm_label IN ('KNOWLEDGE', 'UNCERTAIN')`). Which chunks are auto-mapped vs. escalated to Module D's HITL review is **Module C's decision**, made by C's own boundary / cross-encoder / confidence logic. B's label does not gate what reaches Module D — it's just one input available to C.
+- This keeps recall-first intact end to end: every non-NOISE chunk B produces is consumed and judged by C; nothing is stranded in the queue waiting on a label match.
 
 ## Stability guarantees