Modify parents attribute for MAG ancillary offset files

[mmshaw]

• Note: Don't change the name of the MAG ancillary offset files

• Change the parents attribute of MAG ancillary offset files (one time change; future delivered MAG offset files will have the new versioning scheme)

From Chris Greenaway on 7/24:

MAG L2 is generated with ancillary offset files calculated from a specific version of the L1b/c data. As such the L1 parent data in the L2 is taken from the offset files. If the MAG ancillary offset files do not have their parents updated, then all subsequently generated L2 will not have the correct parents, even if all previous versions were migrated to use the new versioning scheme. We thought this was being done but it seems not having looked at these files.

As far as I can see right now, the parent metadata in the MAG offset files have not been updated to match the new versioning scheme. If they are not updated I expect any reprocessing of current MAG L2 will result in the parents having the old versioning in the metadata. To me, it looks like this is why the L2 data reprocessed at the end of last week had incorrect parents but I could be wrong.

[bryan-harter]

To elaborate:

My understanding is that MAG has custom ancillary CDF files that it custom generates each day, and sends them over to the SDC periodically. Ancillary files are not science files, these are just things like calibration data that imap_processing uses when generating the L1/L2/L3 science files. I'm not clear what specific step in the MAG pipeline uses those ancillary files.

In the global attributes of each of the MAG ancillary files is a "Parents" field, which lists the files that were used to create it. That issue is, that Parents field still contains the files with the version scheme "vNNN" instead of "vMMM.mmmm". While this normally wouldn't be a large issue, it sounds like the problem is that part of the MAG processing reads in the parents from the ancillary files, and just inserts them into the parents attributes for the L1/L2/L3 files.

So we need to do 2 things:

1. Create a script that modifies the "Parents" attributes of these ancillary files, if they are incorrect. I don't know if you'd necessarily have permissions to run it on production, but perhaps you can test it out on dev? And then one of us with elevated permissions could run it.
2. Take a look at the L1/L2/L3 MAG files that have been created recently, and determine if any of them have the incorrect convention in their parents field. I don't know if we necessarily need to modify them or just trigger reprocessing after we change the ancillary files. But it would be good to at least get a list of them.

For reference, we did something similar to this already with the science files: IMAP-Science-Operations-Center/sds-data-manager#1518. "rename.py" is a script that Vineet wrote that opens up CDF files, modifies the attributes, and moves them to a different bucket. It also renames the files, but we do not want to do that with these ancillary files. Other files in the PR allow the script to run remotely on an EC2 and use multiprocessing in python, but I don't know if we'll necessarily need that for this task.

[mmshaw]

@alastairtree tagging you here for awareness

[sapols]

Thanks for elaborating @bryan-harter! I just finished both tasks.

I got my seat on LASP's Claude Team account yesterday so I was able to throw Fable 5 at this problem, and I'm impressed by how well it dug through recent issues/PRs in our repos to understand this stuff. I'll paste the results below (everything below was gathered read-only through the data access API, no bucket access needed).

Results:

First, answering which pipeline step uses these files

To answer your open question, @bryan-harter: since #3288, MAG L2 doesn't just copy the ancillary Parents through; retrieve_mag_l1_inputs_from_l2_offsets (imap_processing/utils.py) reads the offsets file's Parents and downloads the named L1 file as L2's science input, then stamps that name into the L2 product's Parents. And since the migration re-ingested the science archive under new names on 7/25, the old-scheme names in these ancillary files now 404 — so this isn't just cosmetic provenance anymore: MAG L2 processing hard-fails on any day whose offsets file still carries old-scheme Parents. (Offsets files with an empty Parents are fine — the code returns [] and L2 falls back to its passed-in dependencies.)

Task 1: scan of all 577 offsets ancillary files

Status	Count	Data days	Notes
stale (needs fix)	298 (179 norm, 119 burst)	2026-01-01 → 2026-05-31	old-scheme L1C names that now 404
ok	239	2025-11 → 2026-02	already new-scheme — looks like MAG redelivered corrected files for those days
empty Parents	40	2025-09 → 2025-12	harmless (see above)

By latest-version-per-day, 267 mode-days are stale. The mapping is the mechanical vNNN → v001.0NNN (matching the alembic backfill's major_version = 1), and I verified that all 298 rewritten parent sets resolve at the SDC — zero misses — before proposing to write anything.

Full per-file list: offsets_parents_scan.csv

Task 2: audit of recent MAG L1/L2/L3

I audited the *-srf L2 products as a proxy (all five coordinate frames of a mode/day share identical Parents), latest version per day, plus all L1 files ingested after the migration. 389 of 404 recent L2 mode-days have bad parents (multiply by 5 frames for file count), in three distinct populations:

1. 265 with stale old-scheme L1C parents — generated 7/16–7/21 by post-Update MAG l2 code to always retrieve input science files from ancillary #3288 code embedding the offsets' stale Parents verbatim; the migration then renamed these L2 files without rewriting their Parents. Since their provenance is stale either way, triggering reprocessing after the ancillary fix seems like the right remediation rather than patching them.
2. 124 with mangled ancillary parents — these did get their Parents rewritten during the migration, but the rewrite hit ancillary names too: they now reference e.g. imap_mag_l2-calibration_20260301_v001.0006.cdf and imap_mag_l2-norm-offsets_20260428_20260428_v001.0002.cdf — files that don't exist (the real ones are _v006.cdf / _v002.cdf, since ancillary files keep old-scheme naming). This is the mirror image of this issue — I think it deserves its own sds-data-manager issue. It also can't be fixed by the ancillary script here.
3. 15 clean — generated 7/21 post-migration for Oct 2025 days, whose offsets files have empty Parents (the fallback path works).

L1 files generated since the migration are clean (their ancillary calibration parents are old-scheme vNNN, which is correct). L3: no MAG L3 files in the archive yet.

Per-file lists:
l2_srf_audit.csv and l1_recent_audit.csv

Scripts

Both scripts are in this gist: https://gist.github.com/sapols/a9bed63a0569b616766b99f7b30dad71

The fix script (fix_ancillary_parents.py) is modeled on rename.py's Parents branch per your pointer: dry-run by default, rewrites only Parents (filenames, Data_version, Logical_file_id untouched), skips already-correct files, --verify re-checks each rewritten name against the SDC, and same-key upload to a destination bucket (same-bucket overwrite requires an explicit --in-place). I validated the rewrite on a real offsets file and diffed with an independent CDF reader: only Parents changes, all variables/attrs byte-identical. One important difference from rename.py: it only rewrites old-scheme science names — ancillary names (e.g. imap_mag_l1b-calibration_20240229_v001.cdf) legitimately keep vNNN and are left alone (see population 2 in the Task 2 audit above for why that matters).

The audit script (audit_science_parents.py) is what produced the Task 2 CSVs, in case anyone wants to rerun or extend the audit.

I can't run the fix against dev/prod S3 from here, so next step is a dry run by someone with bucket access (both need IMAP_API_KEY set — unauthenticated API queries silently see only publicly-released files). Happy to also PR them into sds-data-manager scripts/ if you'd rather they live in version control for the prod run.

[sapols]

Follow-up on task 2's population 2 above: I opened sds-data-manager#1538 for the mangled-ancillary-parents problem.

I also swept migrated science files across all instruments and levels to gauge the extent; it appears contained to MAG, since only MAG products list ancillary filenames in their Parents at all. Details, mechanism, and suggested remediation are in the issue.