feat: add worker-backed PII local models

[afourniernv]

Overview

Add an implementation-language-neutral inference-provider surface to grpc-v1
workers and use it to back the existing PII redaction local_model lane. The
first optional provider integrates Rampart as an isolated Python worker without
adding ONNX, tokenizer, or model-download dependencies to the Relay host.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

• Add a host-scoped, contract-aware inference-provider registry to Relay core
  and expose provider registration through the Rust and Python worker SDKs.
  Providers are installed before plugin components and deregistered before
  worker shutdown.
• Keep PII policy in the first-party component: content-field targeting,
  compiled JSON paths, payload batching, limits, deadlines, detection
  validation, deterministic composition, replacement, and fail-closed behavior
  remain host-owned.
• Use the active per-call request codec from feat(sanitizers)!: expose per-call codec capabilities #546 when one is available, while
  preserving explicit raw target paths for no-codec calls and omitting unsafe
  event payloads when codec decoding fails.
• Add profile and local_model configuration parity for Rust, Python, Node.js,
  and Go consumers.
• Add an optional manifest-backed Rampart provider. Model files are pinned and
  checksum-verified during an explicit prefetch step; activation is offline,
  and model weights are not bundled.
• Serialize shared ONNX session access, remove cancelled queued calls before
  inference, and send all selected fields through one bounded provider call.
• Add protocol, lifecycle, teardown, cancellation, concurrency,
  malformed-detection, codec, raw-payload, binding, and real-worker end-to-end
  coverage. Update worker and PII documentation.
• No benchmark outputs, evaluation artifacts, dependencies in the Relay host,
  or CI workflow changes are included.

This is additive. Rampart remains optional and disabled by default. It is one
contextual detection lane intended to be composed with deterministic
recognizers, not a standalone privacy guarantee.

Validation completed:

• uv run pre-commit run --all-files
• just test-rust (all PII, core, worker, and completed workspace suites passed; one unrelated CLI path assertion is incompatible with this worktree's symlinked shared target directory)
• just test-python (557 passed)
• just test-python-plugin (125 passed plus real worker E2E)
• just test-node (291 passed)
• just test-go
• cargo test -p nemo-relay-pii-redaction (132 unit and 2 integration tests passed)
• cargo test -p nemo-relay --features worker-grpc --test worker_plugin_integration (33 passed)
• Rampart provider tests (19 passed)
• just docs
• just docs-linkcheck
• cargo fmt --all -- --check
• git diff --check

Local exploratory evaluation on Apple Silicon, using the real manifest-backed
worker and composed deterministic-email plus Rampart profiles:

Selected payload	p50	p95
64 bytes	4.14 ms	4.50 ms
1 KiB	10.32 ms	11.46 ms
8 KiB	76.10 ms	83.60 ms

Cached model initialization was approximately 36 ms with approximately
49.8 MiB additional resident memory. A 64-call concurrent short-input run
completed in 35.97 ms. Longer synthetic samples exposed contextual-name false
negatives, and broad scans produced false positives on some region, UUID, and
trace-like values. Those findings are why the documented configuration composes
the model with deterministic recognizers and restricts selected content fields.

Where should the reviewer start?

Start with crates/core/src/plugin/inference.rs and
crates/core/src/plugin/dynamic/worker.rs for provider ownership, contract
validation, and teardown. Then review crates/pii-redaction/src/local.rs for
the host-owned policy, batching, and codec behavior.
crates/pii-redaction/tests/worker_provider_tests.rs exercises the complete
worker-backed path. The optional Rampart provider is under
crates/pii-redaction/providers/rampart/.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Closes [Enhancement]: Add worker-backed local-model providers for PII redaction #550

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0abf3a47-ce2f-44ef-8d27-4f2c0372ce46

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[afourniernv]

Production-oriented Rampart benchmark rerun on b73ac66f.

Environment

• Apple M4 Pro, 48 GiB RAM, macOS 26.5.2 arm64.
• Real manifest-backed grpc-v1 worker, real pinned Rampart ONNX model, real Relay pii_redaction composition.
• Model was pre-cached and every run used HF_HUB_OFFLINE=1, local_files_only=true, and the pinned model revision.
• More than 12,000 direct and end-to-end operations across payload scaling, concurrent/mixed load, managed LLM request+response sanitization, soak, timeout, kill/restart, and activation/teardown cycles.

Hot-path results

These are two full repeated runs with inference_batch_size=1 and intra_op_threads=8:

Selected content	p50 range
64 B	1.76–2.49 ms
1 KiB	8.42–8.95 ms
8 KiB	70.40–74.21 ms
16 KiB	139.75–143.62 ms
64 KiB	552.94–563.55 ms

The realistic mixed distribution was 50% 128 B, 35% 1 KiB, and 15% 8 KiB:

Caller concurrency	Throughput	p95
1	81.6–83.2 req/s	53.9–54.7 ms
16	77.8–83.6 req/s	142.2–160.6 ms

Throughput stays flat while queueing latency grows, which matches the intentionally serialized detector. This is suitable for small, explicitly selected content fields; it is not a good fit for indiscriminately scanning large event bodies.

Managed LLM request+response sanitization ran 30/30 calls successfully at 19.66 ms p50 / 20.05 ms p95. Every captured event was redacted, the caller-visible response value was preserved, and unselected metadata remained untouched.

Memory and tuning

The current default inference_batch_size=16 is not a production-safe default on this host. After three 64 KiB requests:

Batch size	Worker RSS	Request latency
1	215 MiB	556 ms
2	289 MiB	598 ms
4	441 MiB	600 ms
8	782 MiB	596 ms
16	1,382 MiB	610 ms

Batch size 1 used much less memory and was slightly faster. With that setting, combined host+worker RSS was 184 MiB after activation and 269–271 MiB after the full workload. Two separate 2,000-call soaks added only 0.49–0.67 MiB, so this looks like ONNX arena sizing by batch rather than an unbounded per-request leak.

Cached worker activation was roughly 0.33–0.41 s; teardown p50 was 4.48 ms. The first managed-environment install worked through the normal plugin lifecycle, but required a locally built nemo-relay-plugin 0.7 wheel because that SDK version is not published yet. Publishing it remains a release prerequisite.

Failure behavior

• A killed worker failed closed in 1.62 ms, reported the abnormal teardown, reactivated in 651 ms, and successfully redacted the next request.
• An intentionally undersized deadline failed closed. After the long inference drained, 20–100 ms configurations recovered; 5–10 ms remained below reliable end-to-end overhead.
• All runs ended with zero worker processes left behind.
• The 64-byte/2,000-call soak held p50 at 2.59–2.64 ms and p95 at 2.79–3.15 ms.

Detection-quality probe

This was a small adversarial smoke corpus, not a model-quality benchmark:

• At score 0.4, the model found 29/35 expected entities and left 6/10 technical negatives clean.
• At 0.8, it found 28/35 and left 8/10 negatives clean.
• Known false positives included us-west-2, a UUID, a commit SHA, and 70B.
• The four non-Latin name cases were all missed, consistent with the model card.
• Artificially repeated text also changed detection behavior by length: the same name pattern was found at 64 B and 16 KiB, but missed at 1 KiB and 8 KiB.

The practical boundary remains important: keep the ML provider on explicit content paths and retain deterministic recognizers for structured identifiers. The end-to-end run confirmed that technical metadata such as region and trace ID stayed outside the selected paths.

Readout

The worker/plugin architecture and failure isolation hold up under load. I would change the default inference batch to 1 before treating this as production-ready, document the serialized-throughput and language limits, and tune the score against a representative product corpus. No benchmark code or result artifacts were added to the repository.