Fix PathNotFound error in msexports ETL ingestion on first run

[Michael Flanakin (flanakin)]

Summary

• The "Get Existing Parquet Files" GetMetadata activity fails with PathNotFound when the destination folder doesn't exist yet (e.g. first ingestion for a scope/dataset/month combination). Reservation recommendation exports hit this most often because their destination path adds an extra exportName segment, but any export type can hit it on a first run.
• Fixed in both places this pattern exists: msexports_ETL_ingestion (Microsoft.CostManagement/Exports/app.bicep) and the analytics ingestion ETL pipeline (Microsoft.FinOpsHubs/Analytics/app.bicep).
• Fix: add 'exists' to the GetMetadata activity's fieldList. Per ADF's GetMetadata docs, specifying exists makes the activity return exists: false instead of throwing when the path is missing — "If exists isn't specified in the field list, the Get Metadata activity fails if the object isn't found." The downstream Filter activity's items expression now checks .output.exists before reading .output.childItems, so a missing folder resolves to an empty list instead of propagating the failure.
• This is the same idiom already used elsewhere in this codebase for "path may not exist yet" scenarios: Check Schema in Exports/app.bicep, and both GetMetadata activities in IngestionQueries/app.bicep (one of which — "Get Existing Parquet Files" — already combines exists + childItems exactly like this fix does).

Root cause verification (re: issue #2088 comment)

A prior comment on the issue did solid root-cause analysis but its line numbers had drifted and its proposed fix options were speculative (custom error handling / parent-folder existence checks / wrapping in an If Condition). I re-verified against the current code:

• Microsoft.CostManagement/Exports/app.bicep: GetMetadata activity is at lines 1090-1122, Filter Out Current Exports at 1123-1147 (comment's estimate of ~1090-1122 / ~1123-1147 was accurate).
• Microsoft.FinOpsHubs/Analytics/app.bicep: identical vulnerable pattern confirmed at lines 1668-1707 (comment said ~1667-1706, essentially correct).
• The actual cleanest fix is ADF's documented, built-in behavior for this exact case (exists in fieldList), not a custom If Condition or Web-activity workaround — and this idiom was already present elsewhere in the repo, so this change makes the two vulnerable call sites consistent with the rest of the codebase rather than introducing a new pattern.

Test plan

• bicep build src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep --stdout — builds cleanly
• bicep build src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep --stdout — builds cleanly
• Verified compiled ARM JSON includes "exists" in both fieldList arrays
• pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint" — 3418/3418 passed
• Manual end-to-end validation against a real deployment with no prior export data (not performed in this environment)

Fixes #2088

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

This changes which error the user gets, but the pipeline still fails on a missing folder.

When ingestion_ExecuteETL runs for a folderPath whose ingestion folder does not exist yet, Get Existing Parquet Files now succeeds with exists: false, Filter Out Folders yields an empty array, and then If No Files (below, @equals(length(activity('Filter Out Folders').output.Value), 0)) fires the Files Not Found Fail activity with errorCode IngestionFilesNotFound.

So the cascade into ingestion_ExecuteETL described in #2088 is not actually resolved here - PathNotFound is just replaced by IngestionFilesNotFound, whose message ("Please confirm the folder path is the full path, including the "ingestion" container and not starting with or ending with a slash") points the operator at a path-formatting problem that is not the real cause.

If a first-run / no-data folder should be a no-op, If No Files needs to distinguish "folder absent" (exists == false) from "folder present but empty / path malformed". If failing loudly is still the intent here, it would be worth saying so in the PR description, since the current text implies this hunk fixes the failure.

[Roland Krummenacher (RolandKrummenacher)]

Partly retracting this.

I checked how the pipeline is invoked: trigger_IngestionManifestAdded fires on a blob-created event for manifest.json in the ingestion container (Analytics/app.bicep:682-697) and passes @triggerBody().folderPath. The folder therefore exists by construction whenever this pipeline runs, so there is no first-run "folder not created yet" case on this path the way there is in msexports_ETL_ingestion.

What the hunk does change is which failure surfaces when the folder genuinely is missing — removed between trigger and run, or a folderPath that makes the join(skip(split(...), 1), '/') derivation at line 1662 point somewhere wrong:

• before: Get Existing Parquet Files fails with PathNotFound, Filter Out Folders is skipped, the pipeline fails at activity level
• now: GetMetadata succeeds with exists: false, Filter Out Folders yields empty, If No Files fires the Files Not Found Fail activity with errorCode: IngestionFilesNotFound

I'll take that as an improvement rather than a regression — a defined errorCode is easier to alert on than a raw ADLS error, and the message ("confirm the folder path is the full path, including the ingestion container...") happens to describe the most likely remaining cause given that derivation. No objection to keeping it.

The one thing I'd still ask for is in the description rather than the code. It says the fix is applied "in both places this pattern exists", which reads as though both were failing the same way. They aren't: Exports writes to a destination folder that legitimately may not exist yet, Analytics reads a source folder a manifest just landed in. A sentence noting the Analytics change is diagnostic rather than a fix would stop the next person concluding that a missing ingestion folder is now a no-op there. Non-blocking.

[Roland Krummenacher (RolandKrummenacher)]

Re-reviewed at 73a0076. Both of my threads are settled — the dependency-condition change is a better fix than what I suggested, and the Analytics hunk holds up once you account for how the pipeline is triggered. Details in the threads.

Net change against dev is +5/-3 across two files, and the three Get Existing Parquet Files call sites (Exports, Analytics, IngestionQueries) now use an identical Succeeded + exists + contains shape, which is worth more than the individual hunks.

One non-blocking ask left in the second thread: the description implies both call sites were failing the same way, and they weren't — Exports genuinely hits a missing destination folder on first run, Analytics doesn't. A sentence there would save the next reader the trace. Approving regardless.