feat(aws-healthomics): document WDL engine selection, silent incompatibilities, and how to read lint results

[hmkim]

What

Two additions to the WDL migration SOP, plus a note on reading lint results:

1. Engine selection. The SOP never mentions WDL_LENIENT, even though the CreateAHOWorkflow schema describes it as the option for migrating Cromwell workflows — which is exactly this SOP's subject. Added to Phase 3, together with the limits of what it buys.
2. ## Silent Incompatibilities. Defects that produce no error: the run reaches COMPLETED and the result means something other than what the author intended. No gate in this SOP catches them.
3. Lint results. LintAHOWorkflowDefinition returns "status": "success" for a file that fails to parse. The real verdict is Return code: inside raw_output.

Mostly additive. The only edited lines are Phase 3's step numbering and Done WHEN, plus two bullets in workflow-development.md's Linting section.

1. WDL_LENIENT is missing from the Cromwell migration SOP

migration-guide-for-wdl.md:14 covers WDL versions but not engine choice:

• WDL 1.0+ syntax is required (draft-2 is NOT supported).

Meanwhile the CreateAHOWorkflow tool schema says of its engine parameter:

WDL_LENIENT allows for some WDL directives that don't strictly meet the WDL spec and can be useful when migrating legacy workflows designed to run on Cromwell.

A bridge documented as being for Cromwell migration is absent from the Cromwell migration SOP. An agent following this SOP will register under WDL and may rewrite directives that did not need rewriting.

But introducing it alone would create a new misconception — that lenient gets legacy workflows through. It does not, and I measured the boundary. Identical bytes, identical parameters, engine as the only variable:

engine	registration	run	message
WDL	ACTIVE	FAILED	coercing String to Int: invalid literal for int() with base 10: '3.7'
WDL_LENIENT	ACTIVE	FAILED	cannot coerce String '3.7' to Int: value has a fractional part

(workflows 6243482 / 5645495; runs 2968222 / 4945591)

Different wording, same outcome. WDL_LENIENT's leniency is about registration-time syntax, not runtime type coercion — and note that both engines registered the lossy conversion as ACTIVE. Engine selection is not the gate that catches it; nothing does until a task runs.

A related probe, run today in us-west-2 while preparing this PR: a definition with no version declaration registers ACTIVE under both engines, not FAILED.

$ aws omics create-workflow --engine WDL_LENIENT ...     -> id 8405254, CREATING
$ aws omics get-workflow --id 8405254
{ "engine": "WDL_LENIENT", "status": "ACTIVE",
  "statusMessage": "main.wdl (Ln 1, Col 1) MissingVersion, document should
                    declare WDL version; draft-2 assumed ..." }

Same result under WDL (workflow 3334792). So migration-guide-for-wdl.md:14's "draft-2 is NOT supported" does not mean rejected at creation — the service assumes draft-2 and records it as a warning on an ACTIVE workflow. I have left line 14 alone, since I do not know whether "not supported" is meant as unvalidated-and-unrecommended. But it is why step 4 says neither engine substitutes for declaring the version, rather than claiming lenient rejects draft-2 (an earlier draft of this PR did claim that, and it is wrong).

So Phase 3 step 4 introduces the option and states both limits in the same breath.

2. ## Silent Incompatibilities

The existing Phases fail loudly or not at all. This section collects the cases that pass everything and change the meaning of the result. Nine items, each with a Before/After, a detection rule, or the service's own wording:

• Outputs written outside the task working directory. samtools sort -o /data/out.bam — the task exits 0 and the file is discarded. Data loss, not failure. Distinct from Phase 5, which covers outputs that were never declared; here it is declared and written somewhere that is not collected.
• String used to pass a directory of files. Only File/Directory values are localized. A String is opaque text, so nothing is staged. Classic Cromwell-ism from shared-filesystem days. What makes it expensive to debug is that the error names a file the author never mentioned — a .bwt index — several lines from the declaration that caused it.
• Thread counts not tied to the CPU request. Int threads = 16 with runtime { cpu: 4 }. A literal in a flag is visible on review; a literal in a declaration referenced as -t ~{threads} is not.
• Unbounded scatter. Cromwell deployments were bounded by a cluster queue; there is no equivalent here. Points at run groups, with --max-duration labelled as minutes — the flag takes minutes and that is easy to misread as hours.
• GB where the task was tuned in GiB. Both forms parse and run; they differ by 7.4%, enough to move a borderline task into an OOM kill. A bare memory: 8 is a different bug (it does not reach COMPLETED) so it is pointed back at the Phase 2 audit rather than described here.
• Absolute and remote imports. import "/home/shared/wdl/tasks/align.wdl" resolves on the origin cluster and is not in the zip; an http(s):// import is not fetched. Phase 3 step 3 checks import versions, aliasing and cycles — not whether the path exists in the package. On-prem definitions almost always carry at least one. This item is flagged in the text as failing rather than passing silently, since it surfaces as a registration FAILED at a distance from its cause.

Three of the nine are quoted directly from the AWS docs rather than described in my own words, because they are directives whose HealthOmics meaning differs from their Cromwell meaning and I would rather the SOP carry the service's own wording:

• preemptible — in HealthOmics this controls 5XX retry behavior and has no spot/discounted-capacity meaning at all. The documented values are 0 (opt out), 1, and 2 (retry limit, 2 being the default). So a Cromwell preemptible: 2 — two attempts on preemptible VMs — silently becomes a no-op that still reads like a cost control. (WDL support, "Configure task retry for service errors")
• disks — "The mount path and disk type specifier (SSD, HDD) are ignored — only the numeric size is extracted. If multiple entries are declared, the sizes are summed into a single /tmp allocation." So a scratch layout split across named volumes does not survive migration. Whether the size is used at all depends on scratchStorageMode: in the default SHARED mode disks is ignored entirely for CPU tasks; under LOCAL it is honored as a hint rounded up to 16 GiB. (WDL support, "Supported WDL disks forms"; ephemeral storage)
• maxRetries — retries OOM failures with memory doubling, and "requires GNU findutils 4.2.3+" in the image. A task that declares retries but sits on an image without the package is indistinguishable from one that retried and failed again, so the section gives a find --version check rather than guessing at which base images carry it. (WDL support, "Configure task retry for out of memory")

The disks finding is also why I touched Phase 2 step 2. It currently reads Identify tasks missing cpu, memory, or disks attributes, which puts disks alongside two genuinely required attributes; a missing disks is not a defect. I changed it to note disks without flagging its absence, and pointed at the new section. I did not touch the three added doc links in ## References' neighbours or anything else in Phase 2.

On evidence generally: the engine comparison, the MissingVersion probes, and the lint behavior are mine, measured. The other six items in the new section come from porting Cromwell WDL plus the WDL/HealthOmics docs, and each is stated as a rule rather than as a measurement. Two corrections I made to my own earlier draft, in case they matter to your reading of the rest: --max-duration is minutes (checked against aws omics create-run-group help), and I removed a claim that these workflows have no internet access, since vpc-connected-workflow-runs.md in this same Power says otherwise — the real constraint on imports is that they resolve from the zip package, not from the network. Happy to cut any item you would rather not assert.

3. Reading lint results

workflow-development.md:88-91 says to call the Lint* tools and not deploy if errors exist, but not how to tell. Measured:

--- hello.draft2.wdl ---
  "linter": "miniwdl"
  "status": "success"
  raw_output: Unexpected token Token(COMMAND2_FRAGMENT, ...)
             * Hint: document should begin with WDL version declaration
             Return code: 2

"status": "success" means the linter ran, not that the workflow is valid. An agent that branches on status calls a parse failure a pass. The verdict is Return code: in raw_output.

This is the same envelope-vs-verdict shape as CreateWorkflow returning 200 + CREATING for a definition that later goes FAILED (that one is #177).

I also added a line noting that a clean lint means the definition parses — it is not evidence of semantic correctness — linking to the new Silent Incompatibilities section. Every item in that section lints clean.

Scope

migration-guide-for-wdl.md — Phase 2 step 2 (disks no longer listed as a required attribute), Phase 3 (engine selection, lint verdict, renumbering of the old step 4), the new ## Silent Incompatibilities section, a cross-reference from ## WDL-Specific Considerations, and three doc links in ## References. Plus two bullets in workflow-development.md's Linting section.

Not changed: the Nextflow guide, container/ECR guidance, resource limit values, and migration-guide-for-wdl.md:14 ("draft-2 is NOT supported") — see the note above on why I left that one to you.

Independent of my #177 and #178 — no overlapping hunks, any merge order works. If this is too much for one PR, the natural split is engine-selection + lint (small, measured) from the Silent Incompatibilities section (larger, doc-derived); say the word and I'll split it.

Measurement conditions

The engine comparison: 2026-07, ap-northeast-2, two accounts (one with no pre-existing HealthOmics resources), miniwdl 1.15.0 / Python 3.12.13. The MissingVersion probes (3334792, 8405254) and the lint re-check: 2026-08-04, us-west-2, against awslabs.aws-healthomics-mcp-server@latest. Run and workflow IDs are left unmasked as they are meaningless without the account ID.

[hmkim]

Re-verified the lint behavior against the current awslabs.aws-healthomics-mcp-server@latest just now, since the original measurement was from 2026-07 and I wanted to confirm it had not been fixed in the meantime:

LintAHOWorkflowDefinition(workflow_content=<a definition that does not parse>, workflow_format='wdl')

{
  "status": "success",
  "linter": "miniwdl",
  "raw_output": "... Unexpected token Token('COMMAND2_FRAGMENT', ...)
                  * Hint: document should begin with WDL version declaration
                  Return code: 2"
}

Still "status": "success" over Return code: 2. An agent branching on status treats this as a pass.

[hmkim]

Updated: the three runtime directives I had originally left out — preemptible, disks, maxRetries — are in fact documented, so they are now in the Silent Incompatibilities section with the service's own wording quoted rather than my paraphrase. Sources:

• WDL support in HealthOmics — preemptible (5XX retry opt-out), maxRetries (OOM retry + findutils requirement), supported disks forms
• Ephemeral storage — scratchStorageMode, and when a disks size is honored vs ignored

Two things I corrected in the process, in case they affect how you read the rest:

1. My earlier framing had disks as simply discarded. That is right for the default SHARED mode on CPU tasks, but under scratchStorageMode=LOCAL the size IS honored as a hint rounded to 16 GiB, and GPU tasks always use local NVMe regardless. The section now states both branches.
2. I had written that conda-based biocontainers generally ship findutils 4.2.3+ while slim/Alpine images often do not. That is not in your docs and is not something I should assert in your SOP, so it is replaced with a find --version check.

Also adjusted Phase 2 step 2, which listed disks next to cpu and memory as something to flag when missing — a missing disks is not a defect.

[markjschreiber]

Some great updates! Thanks