From aa236dbebc04a51287eb7e4c732b6284507deba5 Mon Sep 17 00:00:00 2001 From: agentnightshift Date: Sat, 11 Jul 2026 10:59:21 -0500 Subject: [PATCH] feat(memory): provenance metadata and origin filtering Add memory + action lineage tracking (INVEST #3) so agents can distinguish user-asserted facts from agent-inferred ones and audit where an inference came from. Every write now records metadata.provenance, defaulting origin to agent_inferred unless the caller explicitly claims user_asserted, since that claim must never be silently upgraded. search_context gains filter_origin as a post-filter (same pattern as the existing valid_at post-filter) to avoid a new composite Firestore index. Docs missing provenance never match a filter_origin query, which also covers legacy documents until the backfill script runs. Also closes a response-payload gap from the temporal-validity work: fetch/ remember/search responses were missing valid_from, valid_until, supersession_reason, and initiator despite storing them. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 16 +- functions/package.json | 1 + .../scripts/backfill-firestore-provenance.mjs | 172 +++++++++++++++ functions/src/mcpServer.ts | 39 +++- functions/src/service.ts | 37 +++- functions/src/types.ts | 20 ++ functions/test/mcp.integration.test.ts | 205 ++++++++++++++++++ functions/test/service.test.ts | 174 ++++++++++++++- metacortexplan.md | 4 +- 9 files changed, 652 insertions(+), 16 deletions(-) create mode 100644 functions/scripts/backfill-firestore-provenance.mjs diff --git a/CLAUDE.md b/CLAUDE.md index de222f9..7714d82 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,13 +67,13 @@ Auth uses timing-safe token comparison. Origin allowlisting supports `"*"` wildc | Tool | Purpose | |------|---------| -| `remember_context` | Single write tool for chat and admin clients: save durable memory with optional topic, draft flag or explicit branch state, image input, and artifact refs | -| `search_context` | Query → embedding → Firestore vector similarity search (cosine, top-K) with metadata filters | +| `remember_context` | Single write tool for chat and admin clients: save durable memory with optional topic, draft flag or explicit branch state, image input, artifact refs, and provenance metadata | +| `search_context` | Query → embedding → Firestore vector similarity search (cosine, top-K) with metadata and provenance origin filters | | `fetch_context` | Retrieve one stored memory by document ID after search | | `deprecate_context` | Soft-delete: mark document as deprecated, record superseding document ID, and record supersession_reason ("changed" sets valid_until, "corrected" does not) | | `consolidate_context` | Merge N related memories into one canonical active memory via LLM; deprecates all sources with `superseded_by` pointing to the merged result. Defaults to WIP queue for a topic; accepts explicit `source_ids` for targeted consolidation | -`remember_context` also accepts optional `valid_from`/`valid_until` (epoch-ms numbers) so a write can carry its temporal validity window from creation. +`remember_context` also accepts optional `valid_from`/`valid_until` (epoch-ms numbers) so a write can carry its temporal validity window from creation. It also accepts optional `origin`, `source_session`, `derived_from`, and `confidence` for provenance tracking (`origin` defaults to `agent_inferred` when omitted). `search_context` accepts optional `filter_origin` to filter search results by provenance origin as a post-filter (no new Firestore index). ### MCP Prompts @@ -91,19 +91,19 @@ Auth uses timing-safe token comparison. Origin allowlisting supports `"*"` wildc | `errors.ts` | ~9 | `HttpError` exception with `statusCode` field | | `merging.ts` | ~74 | `LlmMergeClient` interface + `GeminiMergeClient` — calls Gemini to merge N memory contents into one | | `runtime.ts` | ~107 | Dependency injection: `createRuntime()` lazily creates and caches Gemini clients, Firestore repo, service | -| `service.ts` | ~401 | `MetaCortexService` — remember/store/search/fetch/deprecate/consolidate flows | +| `service.ts` | ~468 | `MetaCortexService` — remember/store/search/fetch/deprecate/consolidate flows | | `observability.ts` | ~150 | Structured tool-event and request-event logging plus Firestore-backed `memory_events` audit trail | | `embeddings.ts` | ~195 | `GeminiEmbeddingClient` + `GeminiMultimodalPreparer` (image→text normalization for retrieval) | | `normalize.ts` | ~8 | Shared text-normalization helper | | `memoryRepository.ts` | ~228 | Firestore CRUD: `store()`, `search()` (findNearest + cosine), `deprecate()`, `getConsolidationQueue()` | -| `types.ts` | ~138 | Enums (`BRANCH_STATES`, `MEMORY_MODALITIES`, `MCP_TOOL_NAMES`) and interfaces | -| `mcpServer.ts` | ~639 | MCP tool registration with Zod schemas, filtered by client's `allowedTools` and `allowedFilterStates`; also registers the `correct_memory` prompt unconditionally | +| `types.ts` | ~175 | Enums (`BRANCH_STATES`, `MEMORY_MODALITIES`, `MCP_TOOL_NAMES`, `PROVENANCE_ORIGINS`) and interfaces | +| `mcpServer.ts` | ~674 | MCP tool registration with Zod schemas, filtered by client's `allowedTools` and `allowedFilterStates`; also registers the `correct_memory` prompt unconditionally | ### Data Flow -**remember_context**: Chat/admin input → server defaults/inference for metadata and lifecycle state → Gemini multimodal normalization (if image) → canonical `content` + internal `retrieval_text` → Gemini embedding (deployment currently pinned to 768-dim) → Firestore document with vector + metadata +**remember_context**: Chat/admin input → server defaults/inference for metadata (including provenance, where origin defaults to agent_inferred) and lifecycle state → Gemini multimodal normalization (if image) → canonical `content` + internal `retrieval_text` → Gemini embedding (deployment currently pinned to 768-dim) → Firestore document with vector + metadata (including provenance origin) -**search_context**: Query text → Gemini embedding → Firestore `findNearest()` (cosine distance, top-K) with required `branch_state` and optional topic filter; optional `valid_at` post-filters results in the service layer by temporal validity window +**search_context**: Query text → Gemini embedding → Firestore `findNearest()` (cosine distance, top-K) with required `branch_state` and optional topic filter; optional `valid_at` post-filters results in the service layer by temporal validity window, and optional `filter_origin` post-filters by provenance origin **fetch_context**: Document ID → direct Firestore read of one stored memory diff --git a/functions/package.json b/functions/package.json index 96847d0..b7e0bed 100644 --- a/functions/package.json +++ b/functions/package.json @@ -11,6 +11,7 @@ "build": "tsc -p tsconfig.json", "clean": "node -e \"const fs=require('fs'); fs.rmSync('lib',{recursive:true,force:true}); fs.rmSync('coverage',{recursive:true,force:true});\"", "backfill:ttl": "node scripts/backfill-firestore-ttl.mjs", + "backfill:provenance": "node scripts/backfill-firestore-provenance.mjs", "eval:generate": "tsx scripts/retrieval-eval.ts generate-isolated", "eval:import": "tsx scripts/retrieval-eval.ts import-production", "eval:run": "tsx scripts/retrieval-eval.ts run", diff --git a/functions/scripts/backfill-firestore-provenance.mjs b/functions/scripts/backfill-firestore-provenance.mjs new file mode 100644 index 0000000..1c30f61 --- /dev/null +++ b/functions/scripts/backfill-firestore-provenance.mjs @@ -0,0 +1,172 @@ +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +import { getApps, initializeApp } from "firebase-admin/app"; +import { getFirestore } from "firebase-admin/firestore"; + +const scriptDir = path.dirname(fileURLToPath(import.meta.url)); +const functionsDir = path.resolve(scriptDir, ".."); +const repoRoot = path.resolve(functionsDir, ".."); +const explicitEnvKeys = new Set(Object.keys(process.env)); +const loadedEnv = {}; + +for (const fileName of [".env", ".env.prod"]) { + loadEnvFile(path.join(functionsDir, fileName), loadedEnv); +} + +for (const [key, value] of Object.entries(loadedEnv)) { + if (!explicitEnvKeys.has(key)) { + process.env[key] = value; + } +} + +const args = process.argv.slice(2); +const write = args.includes("--write"); +const projectId = + readArg("project") ?? + process.env.GOOGLE_CLOUD_PROJECT ?? + process.env.GCLOUD_PROJECT ?? + readFirebaseProject(repoRoot) ?? + "my-brain-88870"; +const memoryCollection = + readArg("memory-collection") ?? + process.env.MEMORY_COLLECTION?.trim() ?? + "memory_vectors"; +const batchSize = positiveInt(readArg("batch-size"), 250, "batch-size"); + +if (getApps().length === 0) { + initializeApp({ projectId }); +} + +const firestore = getFirestore(); + +console.log(`project: ${projectId}`); +console.log(`mode: ${write ? "write" : "dry-run"}`); +console.log(`collection: ${memoryCollection}`); + +const result = await backfillProvenance( + firestore.collection(memoryCollection), + batchSize, + write +); + +console.log(`${memoryCollection}:`, result); + +if (!write) { + console.log("Dry run complete. Re-run with --write to apply updates."); +} + +async function backfillProvenance(collection, batchLimit, shouldWrite) { + // This legacy backfill reads the collection in one pass; for collections above + // roughly 100k docs, switch to paginated reads with limit/startAfter. + const snapshot = await collection.get(); + const updates = []; + let skipped = 0; + + for (const doc of snapshot.docs) { + const data = doc.data(); + + if (data.metadata && typeof data.metadata.provenance !== "undefined") { + skipped += 1; + } else if (data.metadata && typeof data.metadata.provenance === "undefined") { + updates.push({ + ref: doc.ref, + update: { + "metadata.provenance": { origin: "legacy_import" } + } + }); + } + } + + if (shouldWrite && updates.length > 0) { + await commitUpdates(updates, batchLimit); + } + + return { + scanned: snapshot.size, + update_count: updates.length, + skipped + }; +} + +async function commitUpdates(updates, batchLimit) { + for (let index = 0; index < updates.length; index += batchLimit) { + const batch = firestore.batch(); + + for (const { ref, update } of updates.slice(index, index + batchLimit)) { + batch.update(ref, update); + } + + await batch.commit(); + } +} + +function readArg(name) { + const index = args.findIndex(arg => arg === `--${name}`); + + if (index === -1) { + return undefined; + } + + return args[index + 1]; +} + +function positiveInt(value, fallback, key) { + if (!value) { + return fallback; + } + + const parsed = Number.parseInt(value, 10); + + if (!Number.isInteger(parsed) || parsed <= 0) { + throw new Error(`${key} must be a positive integer`); + } + + return parsed; +} + +function loadEnvFile(filePath, target) { + if (!fs.existsSync(filePath)) { + return; + } + + for (const rawLine of fs.readFileSync(filePath, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + + if (!line || line.startsWith("#")) { + continue; + } + + const separatorIndex = line.indexOf("="); + + if (separatorIndex === -1) { + continue; + } + + const key = line.slice(0, separatorIndex).trim(); + let value = line.slice(separatorIndex + 1).trim(); + + if ( + (value.startsWith("\"") && value.endsWith("\"")) || + (value.startsWith("'") && value.endsWith("'")) + ) { + value = value.slice(1, -1); + } + + target[key] = value; + } +} + +function readFirebaseProject(rootDir) { + const firebaseRcPath = path.join(rootDir, ".firebaserc"); + + if (!fs.existsSync(firebaseRcPath)) { + return undefined; + } + + const firebaseRc = JSON.parse(fs.readFileSync(firebaseRcPath, "utf8")); + const project = firebaseRc.projects?.prod ?? firebaseRc.projects?.default; + + return typeof project === "string" && project.trim() ? project.trim() : undefined; +} diff --git a/functions/src/mcpServer.ts b/functions/src/mcpServer.ts index 75f38e7..3038ff8 100644 --- a/functions/src/mcpServer.ts +++ b/functions/src/mcpServer.ts @@ -18,6 +18,7 @@ import { } from "./service.js"; import { BRANCH_STATES, + PROVENANCE_ORIGINS, SUPERSESSION_REASONS, type BranchState, type McpToolName @@ -76,6 +77,32 @@ export function createMetaCortexMcpServer( .describe( "Optional epoch-ms timestamp marking when this fact stops being valid. Omit for facts with no known end." ), + origin: z + .enum(PROVENANCE_ORIGINS) + .optional() + .describe( + "Optional provenance origin for this write. Defaults to agent_inferred when omitted. Only claim user_asserted when the user explicitly stated this fact themselves." + ), + source_session: z + .string() + .optional() + .describe( + "Optional identifier for the session or conversation this memory was derived from." + ), + derived_from: z + .array(z.string()) + .optional() + .describe( + "Optional list of memory ids that this inference was derived from." + ), + confidence: z + .number() + .min(0) + .max(1) + .optional() + .describe( + "Optional confidence score between 0 and 1 for agent-inferred memories." + ), image_base64: z .string() .optional() @@ -243,6 +270,7 @@ export function createMetaCortexMcpServer( const requestSummary = { topic: normalizeOptionalText(args.topic) ?? "general", branch_state: requestedBranchState, + origin: args.origin, draft: args.draft, content_length: args.content?.trim().length ?? 0, image_present: Boolean(args.image_base64), @@ -300,7 +328,13 @@ export function createMetaCortexMcpServer( valid_at: z .number() .optional() - .describe("Optional epoch-ms timestamp. When provided, only returns memories valid at that point in time (valid_from <= valid_at < valid_until, excluding corrected records).") + .describe("Optional epoch-ms timestamp. When provided, only returns memories valid at that point in time (valid_from <= valid_at < valid_until, excluding corrected records)."), + filter_origin: z + .enum(PROVENANCE_ORIGINS) + .optional() + .describe( + "Optional provenance origin filter. Only returns memories whose provenance.origin matches exactly; memories without provenance metadata are excluded when this filter is set." + ) } }, async args => { @@ -313,7 +347,8 @@ export function createMetaCortexMcpServer( filter_topic: normalizedFilterTopic, filter_state: requestedFilterState, limit: args.limit, - valid_at: args.valid_at + valid_at: args.valid_at, + filter_origin: args.filter_origin }; const result = await observeToolCall( "search_context", diff --git a/functions/src/service.ts b/functions/src/service.ts index 5e9bcbd..bcfde3b 100644 --- a/functions/src/service.ts +++ b/functions/src/service.ts @@ -56,6 +56,13 @@ export class MetaCortexService { }); const now = Date.now(); + const provenance = { + origin: input.origin ?? "agent_inferred", + ...(input.source_session ? { source_session: input.source_session } : {}), + ...(input.derived_from && input.derived_from.length > 0 ? { derived_from: input.derived_from } : {}), + ...(typeof input.confidence === "number" ? { confidence: input.confidence } : {}) + }; + const metadata = { module_name: normalizedModule, branch_state: input.branch_state, @@ -70,7 +77,8 @@ export class MetaCortexService { : {}), ...(typeof input.valid_until === "number" ? { valid_until: input.valid_until } - : {}) + : {}), + provenance } as const; const result = await this.repository.store({ @@ -111,7 +119,11 @@ export class MetaCortexService { image_base64: normalizeOptionalText(input.image_base64), image_mime_type: normalizeOptionalText(input.image_mime_type), valid_from: input.valid_from, - valid_until: input.valid_until + valid_until: input.valid_until, + origin: input.origin, + source_session: input.source_session, + derived_from: input.derived_from, + confidence: input.confidence }); } @@ -134,6 +146,10 @@ export class MetaCortexService { matches = matches.filter(match => matchesValidAt(match.metadata, input.valid_at!)); } + if (input.filter_origin) { + matches = matches.filter(match => match.metadata.provenance?.origin === input.filter_origin); + } + return { matches, appliedFilters: { @@ -408,7 +424,22 @@ function buildPublicMetadata(match: Pick): Record { expect(afterPayload.matches.map((m: any) => m.id)).not.toContain(memoryId); }); + it("remember_context with origin over MCP end-to-end", async () => { + const runtime = createTestRuntime(); + const baseUrl = await startServer( + createMetaCortexApp({ + getConfig: () => runtime.config, + getObserver: () => runtime.observer, + getRuntime: () => runtime + }), + cleanup + ); + + const client = new Client({ + name: "test-client", + version: "1.0.0" + }); + const transport = new StreamableHTTPClientTransport(new URL(`${baseUrl}/mcp`), { + requestInit: { + headers: { + [authorizationHeaderName]: bearerHeader("test") + } + } + }); + + cleanup.push(async () => { + await client.close(); + }); + + await client.connect(transport); + + const rememberResult1 = await client.callTool({ + name: "remember_context", + arguments: { + content: "We are using Ktor for networking.", + topic: "kmp-networking", + origin: "user_asserted" + } + }); + + const rememberPayload1 = parseJsonTextContent(rememberResult1) as any; + expect(rememberPayload1).toMatchObject({ + write_status: "created" + }); + const id1 = rememberPayload1.item.id; + + const fetchResult1 = await client.callTool({ + name: "fetch_context", + arguments: { + id: id1 + } + }); + const fetchPayload1 = parseJsonTextContent(fetchResult1) as any; + expect(fetchPayload1.item.metadata.provenance.origin).toBe("user_asserted"); + + const rememberResult2 = await client.callTool({ + name: "remember_context", + arguments: { + content: "We are using Compose for UI.", + topic: "kmp-ui" + } + }); + + const rememberPayload2 = parseJsonTextContent(rememberResult2) as any; + expect(rememberPayload2).toMatchObject({ + write_status: "created" + }); + const id2 = rememberPayload2.item.id; + + const fetchResult2 = await client.callTool({ + name: "fetch_context", + arguments: { + id: id2 + } + }); + const fetchPayload2 = parseJsonTextContent(fetchResult2) as any; + expect(fetchPayload2.item.metadata.provenance.origin).toBe("agent_inferred"); + }); + + it("search_context with filter_origin over MCP end-to-end", async () => { + const runtime = createTestRuntime(); + const baseUrl = await startServer( + createMetaCortexApp({ + getConfig: () => runtime.config, + getObserver: () => runtime.observer, + getRuntime: () => runtime + }), + cleanup + ); + + const client = new Client({ + name: "test-client", + version: "1.0.0" + }); + const transport = new StreamableHTTPClientTransport(new URL(`${baseUrl}/mcp`), { + requestInit: { + headers: { + [authorizationHeaderName]: bearerHeader("test") + } + } + }); + + cleanup.push(async () => { + await client.close(); + }); + + await client.connect(transport); + + const rememberResult1 = await client.callTool({ + name: "remember_context", + arguments: { + content: "We are using Ktor for networking.", + topic: "kmp-networking", + origin: "user_asserted" + } + }); + const id1 = (parseJsonTextContent(rememberResult1) as any).item.id; + + const rememberResult2 = await client.callTool({ + name: "remember_context", + arguments: { + content: "We are using Ktor for networking also.", + topic: "kmp-networking", + origin: "agent_inferred" + } + }); + const id2 = (parseJsonTextContent(rememberResult2) as any).item.id; + + const searchResult = await client.callTool({ + name: "search_context", + arguments: { + query: "networking", + filter_topic: "kmp-networking", + filter_origin: "user_asserted" + } + }); + + const searchPayload = parseJsonTextContent(searchResult) as any; + const matchIds = searchPayload.matches.map((m: any) => m.id); + + expect(matchIds).toContain(id1); + expect(matchIds).not.toContain(id2); + }); + + it("fetch_context response includes temporal and provenance metadata", async () => { + const runtime = createTestRuntime(); + const baseUrl = await startServer( + createMetaCortexApp({ + getConfig: () => runtime.config, + getObserver: () => runtime.observer, + getRuntime: () => runtime + }), + cleanup + ); + + const client = new Client({ + name: "test-client", + version: "1.0.0" + }); + const transport = new StreamableHTTPClientTransport(new URL(`${baseUrl}/mcp`), { + requestInit: { + headers: { + [authorizationHeaderName]: bearerHeader("test") + } + } + }); + + cleanup.push(async () => { + await client.close(); + }); + + await client.connect(transport); + + const rememberResult = await client.callTool({ + name: "remember_context", + arguments: { + content: "We are using Ktor for networking.", + topic: "kmp-networking", + valid_from: 10000, + valid_until: 20000, + origin: "legacy_import" + } + }); + + const rememberPayload = parseJsonTextContent(rememberResult) as any; + expect(rememberPayload).toMatchObject({ + write_status: "created" + }); + const id = rememberPayload.item.id; + + const fetchResult = await client.callTool({ + name: "fetch_context", + arguments: { + id + } + }); + + const fetchPayload = parseJsonTextContent(fetchResult) as any; + expect(fetchPayload.item.metadata).toMatchObject({ + valid_from: new Date(10000).toISOString(), + valid_until: new Date(20000).toISOString(), + provenance: { + origin: "legacy_import" + } + }); + }); + it("enforces tool scoping on client-specific endpoints", async () => { const runtime = createTestRuntime({ clientProfiles: [ diff --git a/functions/test/service.test.ts b/functions/test/service.test.ts index 17e9852..2afadce 100644 --- a/functions/test/service.test.ts +++ b/functions/test/service.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { MetaCortexService } from "../src/service.js"; +import { MetaCortexService, buildFetchPayload } from "../src/service.js"; import { createTestConfig, FakeMemoryContentPreparer, @@ -332,6 +332,178 @@ describe("MetaCortexService", () => { expect(result.matches.map(m => m.id)).toContain(stored.id); }); + it("storeContext() defaults origin to agent_inferred when not provided", async () => { + const { service } = createService(); + const result = await service.storeContext({ + content: "No origin test content", + module_name: "general", + branch_state: "active" + }); + + expect(result.metadata.provenance).toBeDefined(); + expect(result.metadata.provenance?.origin).toBe("agent_inferred"); + }); + + it("storeContext() preserves explicitly provided origin", async () => { + const { service } = createService(); + const result = await service.storeContext({ + content: "User asserted content", + module_name: "general", + branch_state: "active", + origin: "user_asserted" + }); + + expect(result.metadata.provenance?.origin).toBe("user_asserted"); + }); + + it("storeContext() stores source_session, derived_from, and confidence when provided", async () => { + const { service } = createService(); + const result = await service.storeContext({ + content: "Complete provenance test content", + module_name: "general", + branch_state: "active", + origin: "agent_inferred", + source_session: "session-abc", + derived_from: ["memory-1"], + confidence: 0.8 + }); + + expect(result.metadata.provenance?.origin).toBe("agent_inferred"); + expect(result.metadata.provenance?.source_session).toBe("session-abc"); + expect(result.metadata.provenance?.derived_from).toEqual(["memory-1"]); + expect(result.metadata.provenance?.confidence).toBe(0.8); + }); + + it("storeContext() does not include source_session, derived_from, or confidence if not provided", async () => { + const { service } = createService(); + const result = await service.storeContext({ + content: "Only origin test content", + module_name: "general", + branch_state: "active" + }); + + expect(result.metadata.provenance?.origin).toBe("agent_inferred"); + expect(result.metadata.provenance?.source_session).toBeUndefined(); + expect(result.metadata.provenance?.derived_from).toBeUndefined(); + expect(result.metadata.provenance?.confidence).toBeUndefined(); + }); + + it("searchContext() includes document with matching provenance origin", async () => { + const { service } = createService(); + const stored = await service.storeContext({ + content: "Match origin search content", + module_name: "general", + branch_state: "active", + origin: "user_asserted" + }); + + const result = await service.searchContext({ + query: "search content", + filter_origin: "user_asserted" + }); + + expect(result.matches.map(m => m.id)).toContain(stored.id); + }); + + it("searchContext() excludes document with non-matching provenance origin", async () => { + const { service } = createService(); + const stored = await service.storeContext({ + content: "Non-match origin search content", + module_name: "general", + branch_state: "active", + origin: "agent_inferred" + }); + + const result = await service.searchContext({ + query: "search content", + filter_origin: "user_asserted" + }); + + expect(result.matches.map(m => m.id)).not.toContain(stored.id); + }); + + it("searchContext() excludes documents without any provenance metadata when filter_origin is set", async () => { + const { service, repository } = createService(); + + const legacyStoreResult = await repository.store({ + content: "Legacy doc with Ktor networking", + retrievalText: "Legacy doc with Ktor networking", + embedding: [1, 0, 0, 0, 0, 0], + idempotencyKey: "legacy-fingerprint-123", + metadata: { + module_name: "general", + branch_state: "active", + created_at: Date.now(), + updated_at: Date.now(), + modality: "text" + } + }); + + const normalStoreResult = await service.storeContext({ + content: "Normal doc with Ktor networking", + module_name: "general", + branch_state: "active", + origin: "user_asserted" + }); + + const result = await service.searchContext({ + query: "Ktor networking", + filter_origin: "user_asserted" + }); + + const matchIds = result.matches.map(m => m.id); + expect(matchIds).toContain(normalStoreResult.id); + expect(matchIds).not.toContain(legacyStoreResult.document.id); + }); + + it("buildFetchPayload() exposes valid_from, valid_until, supersession_reason, initiator, and provenance", async () => { + const { service } = createService(); + + const doc1 = await service.storeContext({ + content: "Document with temporal and origin info", + module_name: "general", + branch_state: "active", + valid_from: 1000, + valid_until: 2000, + origin: "user_asserted" + }); + + const doc2 = await service.storeContext({ + content: "Document to deprecate", + module_name: "general", + branch_state: "active" + }); + + const doc3 = await service.storeContext({ + content: "Replacement document", + module_name: "general", + branch_state: "active" + }); + + await service.deprecateContext({ + id: doc2.id, + superseding_id: doc3.id, + supersession_reason: "changed", + initiator: "user" + }); + + const fetch1 = await service.fetchContext({ id: doc1.id }); + const fetch2 = await service.fetchContext({ id: doc2.id }); + + const payload1 = buildFetchPayload(fetch1); + const payload2 = buildFetchPayload(fetch2); + + const item1Metadata = (payload1.item as any).metadata; + expect(item1Metadata.valid_from).toBe(new Date(1000).toISOString()); + expect(item1Metadata.valid_until).toBe(new Date(2000).toISOString()); + expect(item1Metadata.provenance.origin).toBe("user_asserted"); + + const item2Metadata = (payload2.item as any).metadata; + expect(item2Metadata.branch_state).toBe("deprecated"); + expect(item2Metadata.supersession_reason).toBe("changed"); + expect(item2Metadata.initiator).toBe("user"); + }); + it("fetches a stored document by id", async () => { const { service } = createService(); const stored = await service.storeContext({ diff --git a/metacortexplan.md b/metacortexplan.md index 242f8d0..1a7e20a 100644 --- a/metacortexplan.md +++ b/metacortexplan.md @@ -69,7 +69,7 @@ The first hardening release addressed Firestore collection scaling, payload opti * **Effort:** Low-medium. ### 3. Provenance (Memory + Action Lineage) -* **Status:** Proposed (*Added 2026-06 following a design review*) +* **Status:** Implemented 2026-07-11 * **Goal:** Audit memory origin and protect chronology against agent drift (unintended rewrite/reinterpretation of historical priorities). * **Proposal:** Add a `provenance` object to `MemoryMetadata`: * `origin` (`"user_asserted"` | `"agent_inferred"` | `"legacy_import"`) @@ -77,7 +77,7 @@ The first hardening release addressed Firestore collection scaling, payload opti * `derived_from` (optional array of memory document IDs that an inference drew upon) * `confidence` (optional number) - The agent self-reports `origin` on every write. Add an `origin` filter to `search_context`. For action provenance (which principal initiated a lifecycle mutation and the operation's semantics), extend the existing `memory_events` collection rather than introducing new infrastructure, turning it into an authorization-aware audit log. Backfill existing legacy memories with `origin: "legacy_import"`. + The agent self-reports `origin` on every write. Add an `origin` filter to `search_context`. For action provenance (which principal initiated a lifecycle mutation and the operation's semantics), extend the existing `memory_events` collection rather than introducing new infrastructure, turning it into an authorization-aware audit log. Backfill existing legacy memories with `origin: "legacy_import"`. Note that the origin filter on `search_context` is implemented as a post-filter with no new Firestore indexes needed. * **Effort:** Medium. ### 4. Correction as a User-Initiated Action