Skip to content

Fix PathNotFound error in msexports ETL ingestion on first run #2247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Michael Flanakin (flanakin) wants to merge 2 commits into
base: flanakin/v15-prep
Choose a base branch
from
+5 −3
Open

Fix PathNotFound error in msexports ETL ingestion on first run #2247

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4cc21f6
Fix PathNotFound error in msexports ETL ingestion on first run
flanakin Aug 12, 2026
73a0076
Fix dependency condition to fail loudly on genuine GetMetadata errors
flanakin Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,6 +1109,7 @@ resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' existing = {
}
}
fieldList: [
'exists'
'childItems'
]
storeSettings: {
Expand All @@ -1128,14 +1129,14 @@ resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' existing = {
{
activity: 'Get Existing Parquet Files'
dependencyConditions: [
'Completed'
'Succeeded'
]
}
]
userProperties: []
typeProperties: {
items: {
value: '@if(contains(activity(\'Get Existing Parquet Files\').output, \'childItems\'), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'
value: '@if(and(activity(\'Get Existing Parquet Files\').output.exists, contains(activity(\'Get Existing Parquet Files\').output, \'childItems\')), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'
Comment thread
RolandKrummenacher marked this conversation as resolved.
type: 'Expression'
}
condition: {
Expand Down
3 changes: 2 additions & 1 deletion src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1694,6 +1694,7 @@ resource pipeline_ExecuteIngestionETL 'Microsoft.DataFactory/factories/pipelines
}
}
fieldList: [
'exists'

@RolandKrummenacher Roland Krummenacher (RolandKrummenacher) Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes which error the user gets, but the pipeline still fails on a missing folder.

When ingestion_ExecuteETL runs for a folderPath whose ingestion folder does not exist yet, Get Existing Parquet Files now succeeds with exists: false, Filter Out Folders yields an empty array, and then If No Files (below, @equals(length(activity('Filter Out Folders').output.Value), 0)) fires the Files Not Found Fail activity with errorCode IngestionFilesNotFound.

So the cascade into ingestion_ExecuteETL described in #2088 is not actually resolved here - PathNotFound is just replaced by IngestionFilesNotFound, whose message ("Please confirm the folder path is the full path, including the "ingestion" container and not starting with or ending with a slash") points the operator at a path-formatting problem that is not the real cause.

If a first-run / no-data folder should be a no-op, If No Files needs to distinguish "folder absent" (exists == false) from "folder present but empty / path malformed". If failing loudly is still the intent here, it would be worth saying so in the PR description, since the current text implies this hunk fixes the failure.

@RolandKrummenacher Roland Krummenacher (RolandKrummenacher) Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly retracting this.

I checked how the pipeline is invoked: trigger_IngestionManifestAdded fires on a blob-created event for manifest.json in the ingestion container (Analytics/app.bicep:682-697) and passes @triggerBody().folderPath. The folder therefore exists by construction whenever this pipeline runs, so there is no first-run "folder not created yet" case on this path the way there is in msexports_ETL_ingestion.

What the hunk does change is which failure surfaces when the folder genuinely is missing — removed between trigger and run, or a folderPath that makes the join(skip(split(...), 1), '/') derivation at line 1662 point somewhere wrong:

  • before: Get Existing Parquet Files fails with PathNotFound, Filter Out Folders is skipped, the pipeline fails at activity level
  • now: GetMetadata succeeds with exists: false, Filter Out Folders yields empty, If No Files fires the Files Not Found Fail activity with errorCode: IngestionFilesNotFound

I'll take that as an improvement rather than a regression — a defined errorCode is easier to alert on than a raw ADLS error, and the message ("confirm the folder path is the full path, including the ingestion container...") happens to describe the most likely remaining cause given that derivation. No objection to keeping it.

The one thing I'd still ask for is in the description rather than the code. It says the fix is applied "in both places this pattern exists", which reads as though both were failing the same way. They aren't: Exports writes to a destination folder that legitimately may not exist yet, Analytics reads a source folder a manifest just landed in. A sentence noting the Analytics change is diagnostic rather than a fix would stop the next person concluding that a missing ingestion folder is now a no-op there. Non-blocking.

'childItems'
]
storeSettings: {
Expand All @@ -1720,7 +1721,7 @@ resource pipeline_ExecuteIngestionETL 'Microsoft.DataFactory/factories/pipelines
userProperties: []
typeProperties: {
items: {
value: '@if(contains(activity(\'Get Existing Parquet Files\').output, \'childItems\'), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'
value: '@if(and(activity(\'Get Existing Parquet Files\').output.exists, contains(activity(\'Get Existing Parquet Files\').output, \'childItems\')), activity(\'Get Existing Parquet Files\').output.childItems, json(\'[]\'))'
type: 'Expression'
}
condition: {
Expand Down