fix(search): contain faults while copying results

[aaaaaandrew]

Summary

• guard the final FileMatch copy, where mmap-backed slices are accessed after the per-shard search guard has returned
• drop only matches whose copy faults and keep the response explicitly incomplete
• preserve exact shard provenance through aggregation with one mmap pointer range per shard result
• count each faulted shard once across files, repositories, and streaming events
• schedule the conditional repair path introduced by the parent PR

This is stacked on #1125 and addresses the remaining late-copy fault window described in #1106. Until #1125 merges, GitHub's comparison against upstream main includes both commits; afterward this PR reduces to the result-copy commit.

Behavior

Zoekt intentionally delays copying mmap-backed match data until after aggregation and result limiting. The guard remains at that boundary, so candidates discarded by global limits are not copied.

Each match is copied under a narrowly scoped debug.SetPanicOnFault guard. If an mmap read faults, the unsafe match is dropped and healthy matches remain. Stats.Crashes and the crash metric increment once per faulted shard for the whole request, including StreamSearch results split across multiple repository events.

Repository identity is not used for attribution because one repository can span shards. Before aggregation, each shard result records the address range of its mmap-backed match data. Aggregation and tail truncation preserve those slice pointers, allowing the final copy to resolve the exact searched shard. Overlapping or missing provenance is reported but never used to guess a repair target.

A concurrent watcher replacement unregisters the old shard instance, so a late copy fault from the old snapshot cannot reopen the new generation.

Controlled before/after

The Linux control supplies the final copy function with FileMatch values backed by a truncated MAP_SHARED mapping.

• parent PR without this change: exits on SIGBUS with fatal error: fault
• this PR: drops the unsafe matches, retains healthy matches, records one shard crash, and schedules one repair

Coverage includes Search, StreamSearch, display limiting, a repository split across shards, multiple repository events from one shard, and deliberately ambiguous provenance.

The limited-result benchmark remains near the parent branch (~225-258µs versus ~221-251µs over five 30-iteration samples on darwin/arm64). This is containment for the EFS/NFS failure documented in #1109; immutable shard generations remain the root publication-lifetime fix.

Test plan

• go test ./...
• go test -race ./search
• go vet ./...
• Linux focused fault suite repeated 10 times
• full Linux go test ./search
• BenchmarkShardedSearchManyChunks/max-docs-8

[keegancsmith]

I've never seen this happen in practice. Are you seeing it cause of NFS? I wonder if there is a more fundamental change we can make such that we can treat the shards as proper immutable blobs that we can do a bit of validation on when we load it up.

I have a feeling for a normal FS if we successfully find a match, this part of the code should continue to work.

[keegancsmith]

a repo can be in multiple shards.

[aaaaaandrew]

Right. I removed repository-based attribution. Each shard result now records one range covering its mmap-backed match pointers; the final copy resolves the exact searched shard after aggregation. Overlapping ranges are treated as ambiguous and do not repair either shard. There is a regression with the same repository present in two shards.

[keegancsmith]

definition of Stats.Crashes is the number of shards that crashed, this will increment per file that failed.

[aaaaaandrew]

Agreed. Crash deduplication now lives in the request-scoped provenance tracker, so multiple bad matches and multiple StreamSearch repository events from one shard contribute exactly one crash and one metric increment. Both cases are covered.

[aaaaaandrew]

On the broader question: yes, the observed failure is AWS EFS over NFSv4.1; #1109 has the production sequence and the two-client control. I agree immutable generation paths are the root fix. Load-time validation alone cannot catch this case because the shard loads successfully and its backing object is invalidated later. This PR is the narrow containment for the remaining copy window while that publication-lifetime issue is separate.