emit OTel spans for memory operations (Go + Python ADK, controller DB)#2280
Open
henrikrexed wants to merge 5 commits into
Open
emit OTel spans for memory operations (Go + Python ADK, controller DB)#2280henrikrexed wants to merge 5 commits into
henrikrexed wants to merge 5 commits into
Conversation
…gent-dev#1909) Add dedicated memory.* spans to the Go ADK memory service, alongside the existing gen_ai.* spans on invoke_agent: memory.read (operation=prefetch) - recall; child of invoke_agent memory.write (operation=save) - AddSessionToMemory and the save_memory tool's live write path memory.consolidate (operation=extract) - LLM fact extraction Each span carries a memory.* governance vocabulary (operation/scope/source/ index_ref/injection_result/item.count) plus a memory.sut.* descriptor for the pgvector backend. kagent emits only attributes it can populate truthfully; status/authority/record-id-on-write stay reserved. Additive; exports only when OTEL_TRACING_ENABLED=true. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…time Mirror the Go ADK memory instrumentation in the Python runtime's memory service (_memory_telemetry.py): memory.read (prefetch, with injection_result + item.count), memory.write (save; source=user for the explicit save_memory / add_memory tool path, source=agent_inference for summarized session ingestion), and memory.consolidate (extract). Same SUT descriptor (kagent/vector/pgvector) and reserved-vocabulary discipline as the Go side. Instruments the live write path (add_memory), which the save_memory tool calls, not just add_session_to_memory. Adds opentelemetry-api as a direct dependency plus unit tests asserting the read/write spans and attributes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… child spans Instrument the controller's memory data-access layer (client_postgres.go) so memory searches/inserts/lists/deletes emit db.memory.* child spans under the active request span. Attributes follow DB semantic conventions and are cardinality- and PII-guarded: no raw embedding vectors or query/content text are placed on span attributes. Adds a test asserting the search span and its non-PII attributes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ace propagation Make the a2a-SDK and httpx-client auto-instrumentations opt-out via helm (otel.tracing.a2aSdkInstrumentation / httpxClientInstrumentation, both default true for upstream parity), forwarded to agent pods as OTEL_INSTRUMENTATION_A2A_SDK_ENABLED / _HTTPX_CLIENT_ENABLED. Enrich the delegation execute_tool span with a2a.* attributes (subagent, context lineage, task id/state). Trace continuity is preserved even when the instrumentations are disabled: a request event-hook and the A2A sub-agent interceptor re-inject W3C trace context (correlation headers only, no extra spans) so agent->controller memory calls and agent->agent hops stay in one trace. Adds unit tests for the delegation attributes and the trace-context propagation hooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds OpenTelemetry instrumentation for kagent’s memory subsystem across both agent runtimes (Go + Python ADK) and the controller’s pgvector-backed DB layer, while also ensuring cross-service trace continuity when auto-instrumentations are opted out via Helm/env.
Changes:
- Emit
memory.*spans (and supporting attributes) from Go ADK + Python ADK memory operations, plus A2A delegation span annotations. - Emit controller-side
db.memory.*spans for pgvector CRUD paths with tests guarding against PII/high-cardinality leakage. - Add Helm-configurable opt-out toggles for a2a-SDK and httpx client auto-instrumentations, plus explicit W3C trace-context reinjection hooks and accompanying tests/docs.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| python/uv.lock | Adds opentelemetry-api to Python lockfile dependency set. |
| python/packages/kagent-core/tests/test_tracing_propagation.py | Tests outbound httpx trace-context injection hook. |
| python/packages/kagent-core/tests/test_tracing_configure.py | Tests default-on + opt-out behavior for httpx client instrumentation and FastAPI instrumentation kwargs. |
| python/packages/kagent-core/src/kagent/core/tracing/_utils.py | Adds env-gated httpx client instrumentation (default enabled). |
| python/packages/kagent-core/src/kagent/core/tracing/_propagation.py | New reusable httpx request hook to inject W3C trace context. |
| python/packages/kagent-core/src/kagent/core/tracing/init.py | Exposes inject_trace_context from the tracing package. |
| python/packages/kagent-adk/tests/unittests/test_remote_a2a_tool.py | Verifies A2A subagent interceptor injects traceparent when a span is active. |
| python/packages/kagent-adk/tests/unittests/test_memory_spans.py | New tests for Python ADK memory.* spans/attributes (incl. embed child spans). |
| python/packages/kagent-adk/tests/unittests/test_a2a_telemetry.py | New tests for A2A delegation attribute stamping on existing spans. |
| python/packages/kagent-adk/src/kagent/adk/_remote_a2a_tool.py | Injects W3C trace context for remote-agent delegation + stamps a2a.* attrs onto active span. |
| python/packages/kagent-adk/src/kagent/adk/_memory_telemetry.py | New Python ADK helper module for memory.* span creation/attributes. |
| python/packages/kagent-adk/src/kagent/adk/_memory_service.py | Wraps save/search/session-ingest paths with memory.* spans and embed child spans. |
| python/packages/kagent-adk/src/kagent/adk/_a2a.py | Attaches trace-context injection hook to controller-bound httpx client. |
| python/packages/kagent-adk/src/kagent/adk/_a2a_telemetry.py | New helper module to annotate delegation attributes on the active execute_tool span. |
| python/packages/kagent-adk/pyproject.toml | Declares opentelemetry-api dependency for ADK span instrumentation. |
| helm/kagent/values.yaml | Documents memory spans + adds tracing toggles for a2a-SDK and httpx client instrumentation. |
| helm/kagent/templates/controller-configmap.yaml | Forwards OTEL_INSTRUMENTATION_* env vars into agent pods via controller config. |
| go/core/internal/database/memory_trace_test.go | New tests asserting db.memory.* spans emit and do not leak sensitive attributes. |
| go/core/internal/database/client_postgres.go | Adds db.memory.{search,insert,list,delete} spans around memory CRUD operations. |
| go/adk/pkg/telemetry/memory.go | New Go ADK helper for starting memory.* spans with standardized attributes. |
| go/adk/pkg/memory/save_memory_tool.go | Refactors tool path to call a span-bearing service method. |
| go/adk/pkg/memory/kagent_service.go | Adds Go ADK memory span instrumentation for read/write/consolidate paths. |
| go/adk/pkg/memory/kagent_service_spans_test.go | New Go ADK unit tests asserting memory spans + key attributes. |
| docs/verification/kagent-baseline.dql | Dynatrace query for pre-change “no memory spans” baseline verification. |
| docs/verification/kagent-after.dql | Dynatrace query for post-change memory span inventory verification. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+124
to
+127
| def record_span_error(span: trace.Span, err: Exception) -> None: | ||
| """Mark a span as failed and record the exception on it.""" | ||
| span.record_exception(err) | ||
| span.set_status(trace.Status(trace.StatusCode.ERROR, str(err))) |
Comment on lines
+3
to
+8
| With httpx client auto-instrumentation disabled by default to cut | ||
| span noise, nothing injects W3C trace context (``traceparent`` / ``tracestate``) | ||
| on outbound agent->controller calls. As a result the controller's | ||
| ``/api/memories/*`` work (and any other call made through the agent's shared | ||
| httpx client) starts a *new root trace* instead of nesting under the active | ||
| ``memory.read`` / ``memory.write`` span. |
Comment on lines
+108
to
+110
| if not self._embedding_client: | ||
| logger.warning("No embedding client available for session %s", session.id) | ||
| return |
Comment on lines
+113
to
+115
| if not vectors: | ||
| logger.warning("Failed to generate embeddings for session %s", session.id) | ||
| return |
Comment on lines
+180
to
+182
| if not self._embedding_client: | ||
| logger.warning("No embedding client available") | ||
| return |
Comment on lines
+185
to
+187
| if not vector: | ||
| logger.warning("Failed to generate embedding for memory content") | ||
| return |
Comment on lines
+37
to
+41
| // Span attribute keys (OTel DB semantic conventions + kagent memory extensions). | ||
| const ( | ||
| attrDBSystemName = "db.system.name" | ||
| attrDBOperationName = "db.operation.name" | ||
| attrDBCollectionName = "db.collection.name" |
Comment on lines
+331
to
+334
| ctx, span := telemetry.StartMemorySpan(ctx, telemetry.SpanMemoryConsolidate, | ||
| telemetry.MemoryOperationExtract, telemetry.MemoryScopeUser, s.agentName) | ||
| defer span.End() | ||
|
|
Resolve the 8 Copilot review comments on kagent-dev#2280: - _memory_telemetry.py: import Status/StatusCode from opentelemetry.trace explicitly in record_span_error (robust across otel-api versions). - _memory_service.py: set memory.item.count=0 on all four early-return paths (missing embedding client / empty vectors) so the attribute contract stays consistent for dashboards. - _propagation.py: correct the module docstring — httpx client auto-instrumentation is enabled by default and opt-out, not disabled by default. - client_postgres.go: use semconv v1.39.0 constants/helpers (DBSystemNamePostgreSQL, DBOperationName, DBCollectionName) for the db.* attributes instead of raw string keys, matching the controller's existing semconv usage. Attribute keys/values are byte-identical. - kagent_service.go: set memory.item.count before every return in the memory.consolidate path and record errors on the generation-error return via recordSpanError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR description
Closes #1909
Why
kagent's platform-level OpenTelemetry is already strong: the controller's
invoke_agentspan carries thegen_ai.*GenAI semconv attributes, and HTTP auto-instrumentation covers the session/task endpoints. But the memory subsystem is a blind spot — long-term memory (pgvector-backed) reads, writes, and consolidation emit no dedicated spans, so operators can't see when an agent recalled or stored a memory, how many items came back, or whether a recall was filtered out by the similarity gate. Memory is increasingly where agent behaviour is decided, so this is exactly the layer teams need to observe and, eventually, govern.This PR closes that gap for both agent runtimes (Go ADK and Python ADK), plus the controller's data-access layer, and does it purely additively — no behaviour change, no new runtime dependencies beyond
opentelemetry-api, and everything exports only whenOTEL_TRACING_ENABLED=true.What changed
Memory spans (children of the existing
invoke_agenttrace, so the tree stays connected with thegen_ai.*spans you already see):memory.operationmemory.readsearch_memoryprefetch(child ofinvoke_agent)memory.writesave_memorytool's live write path + session ingestionsavememory.consolidateextractgo/adk/pkg/memory,go/adk/pkg/telemetry/memory.go) and Python ADK (_memory_service.py,_memory_telemetry.py) emit the same convention.go/core/internal/database) emitsdb.memory.*child spans, cardinality- and PII-guarded (never puts raw embedding vectors or query/content text on attributes).Attribute convention — each span carries
memory.operation,memory.scope,memory.source,memory.index_ref,memory.injection_result(injected/filteredfrom the min-score gate),memory.item.count, and amemory.sut.{name,architecture,store_backend}descriptor (kagent/vector/pgvector). Only attributes kagent can populate truthfully today are emitted; a governance vocabulary (memory.status,memory.authority,promote/revoke) is intentionally reserved, not stubbed, until the subsystem models memory lifecycle and injection authority.Trace hygiene / configurable auto-instrumentation — the a2a-SDK and httpx-client auto-instrumentations become opt-out via helm (
otel.tracing.a2aSdkInstrumentation/httpxClientInstrumentation, both defaulttruefor parity), forwarded to agent pods asOTEL_INSTRUMENTATION_*env. The delegationexecute_toolspan is enriched witha2a.*attributes (subagent, context lineage, task id/state). Trace continuity is preserved even with the instrumentations disabled: request event-hooks re-inject W3C trace context (correlation headers only, no extra spans), so agent→controller memory calls and agent→agent hops stay in one trace.Docs for the new span names live under
helm/kagent/values.yaml(otel.tracing).Tests
go/adk/pkg/memory(read span injected/filtered + attributes; write span on the tool path) andgo/core/internal/database(asserts thedb.memory.searchspan fires and never leaks embedding vectors / raw text).go build+go vetclean.ruffclean): memory read/write spans + attributes, a2a delegation attributes, remote-a2a trace continuation, and outbound trace-context propagation, using in-memory span exporters.memory.read/memory.writeland from both the Go and Python runtimes with the expected attributes and correct parent-span nesting (memory.readunderinvoke_agent;memory.writeunder thesave_memoryexecute_toolspan).What this unblocks
memory.*vocabulary gives a forward-compatible contract; when kagent models memory lifecycle/authority, those attributes drop onto the same spans with no span-name churn.