fix(data-package): survive re-uploads and honor client sha256 in server-wrap path#4
Merged
Merged
Conversation
`data_packages.filename` has a UNIQUE constraint. ATAK reuses the same filename (photo capture timestamp) whenever it re-uploads a data package — e.g. re-sending the same photo to a different recipient. On the second and later attempts, `save_data_package_to_db` raised `IntegrityError` on the filename constraint, caught it, and returned a Flask response tuple that its caller ignored. The upload endpoint returned 200 anyway, with a URL containing the new hash. ATAK then sent a CoT referencing that URL, and every recipient's `/Marti/sync/content?hash=<newhash>` lookup returned 404: the file was on disk (keyed by hash) but no `data_packages` row existed for it, so the download handler's DB-first query missed. Silent delivery failure to every recipient after the first. Fix: SELECT-then-INSERT-or-UPDATE. Look up the existing row by filename; if it exists, update its hash and metadata in place; if not, INSERT as before. Latest upload wins, which matches ATAK's mental model when a user re-shares the same photo. The remaining IntegrityError catch handles the narrow race where a concurrent upload of the same filename slips between our SELECT and COMMIT. Reproduced on the operator's Azure OTS install: photo `20260707_214621.jpg.zip` uploaded four times with distinct hashes; DB only stored the first, recipients' `sync/content` requests for attempts 2–4 all 404'd. Also removes the misleading `"This data package has already been uploaded"` response — the endpoint was already returning 200 anyway, so the error tuple was unreachable in practice. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TX-RX
marked this pull request as draft
July 8, 2026 17:27
ATAK computes the sha256 of the raw file it uploads and passes it as
the ?hash= query param on POST /Marti/sync/missionupload. It then
looks up the wrapped package via PUT /Marti/api/sync/metadata/<hash>/tool
and GET /Marti/sync/content?hash=<hash> using that same client hash.
The server was computing its own sha256 of the server-wrapped zip, saving
the file and DB row under that server hash. Every client lookup then 404'd
because the client hash was never on disk. Fileshare CoT reached the
recipient but the follow-up download stalled with a 502/404.
Honor request.args.get('hash') whenever present (which is always, in the
request-driven code path); fall back to computing one only for non-request
callers (there are none today, but the function still accepts a str path).
Also emit a debug log with the filename/hash pair so future upload traces
show the invariant at a glance.
Verified synthetically on Azure OTS 20.114.34.147 at 2026-07-11 17:38 UTC:
raw 2 KB JPG posted with a known sha256 => server saved the wrapped zip
under exactly that hash (log, on-disk file, and HTTP response body all
agreed). Real ATAK 5.7 and iTAK clients pre-wrap and therefore hit
save_data_package_file instead; this branch protects the legacy raw-file
upload path and any third-party tooling that hits missionupload with a
non-zip file.
TX-RX
changed the title
TX-RX
marked this pull request as ready for review
July 11, 2026 17:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related bugs in the data-package upload path
Bug 1 — Re-upload of the same filename kills the transaction
data_packages.filenamehas a UNIQUE constraint. ATAK reuses the same filename (photo capture timestamp) whenever it re-uploads a data package — e.g. re-sending the same photo to a different recipient. The old codeINSERT'd unconditionally insave_data_package_to_db(), so the second attempt raisedIntegrityError, killed the transaction, and 404'd the follow-up metadata PUT. The receiver's fileshare stalled.Fix (5ff2b1a): upsert on filename — SELECT existing row, update in place, or INSERT. Latest hash wins.
Bug 2 — Server-wrap path used server-computed hash, not client-computed
In
create_data_package_zip, the server computed sha256 of the server-wrapped zip and used that for both the on-disk filename and the DB row. But ATAK looks up the wrapped package viaPUT /Marti/api/sync/metadata/<hash>/toolandGET /Marti/sync/content?hash=<hash>using the sha256 it computed of the raw file it sent (also passed as?hash=…on the POST). Server-hash ≠ client-hash → every downstream lookup 404'd, and the fileshare stalled after a successful upload.Fix (1b8d709): honor
request.args.get("hash")when present; fall back to server-computed only for non-request callers (there are none today, but the function accepts astrpath).Verification
Both fixes deployed to Azure OTS since 2026-07-10 22:47 UTC. On 2026-07-11:
IntegrityError, zeroTraceback.create_data_package_zip(both pre-wrap to.zip), so Bug 2 was validated synthetically: a 2 KB test JPG POSTed to/Marti/sync/missionupload?hash=<sha>&filename=test.jpgproduced the log linecreate_data_package_zip - Wrapped and saved data package: test - <sha>where<sha>matches the client's, saved on disk under<sha>.zip, and returned the metadata URL with that same hash. Ironclad proof of the invariant.Files changed
opentakserver/blueprints/marti_api/data_package_marti_api.py(+47 / -28)