L2 semantic-hash predicate for cross-bundle clone detection

[pedroanisio]

Context

Today every L2 chunk carries a cbm:contentSha256 predicate — a literal-bytes hash. Two functions that are byte-different but semantically identical (e.g. one has different whitespace or a renamed local variable) hash to different values.

This blocks an MCP query class: cross-bundle clone detection. We cannot answer "find this function in another repo" or "did this method get moved unchanged across commits."

Borrowed concept from the code-block-migrate bundle: it computes semantic_hash per block and exposes find_blocks_by_semantic_hash / find_semantic_duplicates. The concept is what's valuable; their Rust implementation is not portable to us.

Scope

Extend plugins/chunks_embeddings/ to emit a cbml2:semanticHashLexical predicate on every chunk, computed from a normalized form of the source.

Normalization (conservative — start here)

For all languages: strip line/block comments, collapse runs of whitespace, normalize line endings. SHA-256 over the result.

Optional second variant (separate triple)

cbml2:semanticHashCanonical — Python-only via ast.parse → ast.unparse (canonical formatting) plus optional locals-renaming. Land this as a follow-up after lexical version stabilizes.

Subtasks

• Add _compute_semantic_hash(chunk_text, language) helper in plugins/chunks_embeddings/chunker.py
• Extend plugins/chunks_embeddings/graph_writer.py to emit cbml2:semanticHashLexical
• Add SHACL shape for the new predicate
• Add MCP tool semantic_duplicates(chunk_idx, scope="bundle"|"global") in frontend/mcp_server/ that does SELECT ?other WHERE { ?other cbml2:semanticHashLexical ?h . ?chunk cbml2:semanticHashLexical ?h }
• Add tests/verify_semantic_hash.py:
  • Same source, different whitespace → same hash
  • Same source, different comments → same hash
  • Renamed identifier → different hash (lexical variant is conservative)
  • The 4 byte-identical XSDs in bridge/v1_0/ and earlier duplicated paths (see static/schemas/) → all match

Ground-truth test fixture (already on disk)

The static/schemas/ directory already contains a perfect trivial-case fixture: before deduping, the bundle had 4 byte-identical copies of ddd-python-bridge-spec.xsd. A future expansion can re-add them under tests/fixtures/clones/ for a "must catch byte-identical" assertion.

Acceptance

After this lands, running our pipeline over a repo with two byte-different-but-whitespace-only-divergent functions surfaces them as semantic duplicates via the new MCP tool.

Cost estimate

~150 lines of plugin code + ~50 lines of test + SHACL shape addition. Bounded.