From d242e9030008fe4b9959f9c9b0fe84a09ca72a70 Mon Sep 17 00:00:00 2001 From: Brandon Date: Tue, 4 Aug 2026 14:29:04 +0000 Subject: [PATCH] fix(aws-healthomics): reconcile WDL Phase 1 container strategy with the ECR SOP migration-guide-for-wdl.md Phase 1 required "Zero references to external registries remain" as a completion criterion, but ecr-pull-through-cache.md states that migrating workflows SHOULD use a container registry map (preferred) precisely so container URIs do NOT change. A map leaves the public URIs in the WDL, so the migration SOP defined the preferred approach's end state as failure. Phase 1 also prescribed hand-written docker pull/push scripts without referencing the eight container MCP tools documented in the same Power. Make the approach an explicit per-container choice, route staging through the MCP tools, link the ECR SOP, and split the Done WHEN criteria so each approach has its own. Add a transport-vs-identity decision rule to Technical Patterns: a map is right where the same repository and tag is served from private ECR, and the WDL must be edited where the repository changes, since the command block's correctness then depends on which image is pulled. Show both outcomes in the Container Runtime pattern, which previously showed only the URI-replacement result. Also replace "At least 5 key containers verified accessible from ECR" with per-task command coverage. Accessibility is not containment: an image can exist, pass describe-images, miniwdl check and CreateWorkflow, and still lack a tool the command block pipes to, which surfaces only as "command not found" once the task runs. --- .../steering/migration-guide-for-wdl.md | 76 ++++++++++++++----- 1 file changed, 57 insertions(+), 19 deletions(-) diff --git a/aws-healthomics/steering/migration-guide-for-wdl.md b/aws-healthomics/steering/migration-guide-for-wdl.md index 70c7f596..d5a5f553 100644 --- a/aws-healthomics/steering/migration-guide-for-wdl.md +++ b/aws-healthomics/steering/migration-guide-for-wdl.md @@ -24,31 +24,44 @@ AWS HealthOmics requires: ### Phase 1: Container Inventory and Migration -**Objective**: Identify all containers and create ECR migration plan. +**Objective**: Identify all containers and make them available to HealthOmics from private ECR. **Steps**: 1. Extract all unique container URIs from runtime sections: - Scan all WDL files for `docker:` and `container:` runtime attributes. - Check imported WDL files and sub-workflows. - Identify containers in struct/object definitions. -2. Generate `container_inventory.csv` with columns: Task name, Original container URI, Container registry, Tool name and version, Target ECR URI. -3. Create `scripts/migrate_containers_to_ecr.sh` to: - - Find or create ECR repositories for each tool with access policies allowing the omics principal to read. - - Pull each container from source registry ensuring x86 containers are pulled. - - Tag for ECR: `.dkr.ecr..amazonaws.com//:` - - Push to ECR repositories. -4. Create `scripts/update_container_refs.sh` to: - - Replace all container URIs in WDL task runtime sections. - - Update to use ECR registry. - - Parameterize container references. -5. Create `healthomics.inputs.json` with ECR registry base path parameter. +2. Generate `container_inventory.csv` with columns: Task name, Original container URI, Container registry, Tool names and versions, Target ECR URI, Approach (`registry-map` or `uri-replacement`). +3. For each container, CHOOSE one approach — see **Registry Map or URI Replacement** under Technical Patterns: + - **Registry map (PREFERRED)**: the WDL keeps its original public URIs and HealthOmics redirects them to your pull-through caches. Use this WHERE the image itself is unchanged — same repository, same tag, served from your private ECR instead of the public registry. + - **URI replacement**: edit the WDL. Use this WHERE the migration changes which repository a task pulls from, because the command block's correctness then depends on which image is used. +4. Stage the containers using the MCP tools — PREFER these over hand-written `docker pull`/`docker push` scripts. Follow the [ECR Pull Through Cache SOP](./ecr-pull-through-cache.md): + - Call `ListPullThroughCacheRules` first. IF a valid cache already exists for an upstream registry, reuse it — DO NOT create another. + - Call `CreatePullThroughCacheForHealthOmics` for each remaining upstream registry (`docker-hub`, `quay`, `ecr-public`). This also sets the registry permissions policy and repository creation template that HealthOmics needs. + - Call `CheckContainerAvailability` with `initiate_pull_through: true` to populate and confirm each image. + - Call `CloneContainerToECR` for registries NOT supported by pull-through cache. + - IF using a registry map: call `CreateContainerRegistryMap`, then pass it to `CreateAHOWorkflow` via `container_registry_map` (or `container_registry_map_uri`). + - IF replacing URIs: update the `docker`/`container` values in the WDL task runtime sections, and PREFER parameterizing the registry base path. + + > **The registry map is a workflow attribute, not a run parameter.** It is supplied at workflow creation, and registration does not appear to validate the definition against it: a workflow whose tasks name public registries registers as `ACTIVE` and is unrunnable. The run then fails partway through with `has an invalid structure. Provide a valid ECR image URI`, naming the task it reached rather than the missing attribute, after earlier tasks have already consumed billed compute. ENSURE the map is attached at creation — a successful `CreateAHOWorkflow` does NOT confirm it. +5. Verify container CONTENTS, not just availability. For each task: + - List the binaries its `command` block invokes. Take the first bare word of each pipeline stage and of each line, including inside `if`/`for` bodies, and discard shell builtins (`set`, `cd`, `echo`, `if`, `then`, `fi`, `for`, `do`, `done`, `mkdir`, `mv`, `cp`) and `~{}` interpolations. + - Confirm each one is present in that task's image: + ```bash + docker run --rm sh -lc 'command -v bwa samtools bcftools' + ``` + - IF any binary is absent, the task WILL fail at run time — resolve it via step 6 before proceeding. + + > No static check substitutes for this. `CheckContainerAvailability`, `aws ecr describe-images`, `miniwdl check`, and `CreateWorkflow` all pass for an image that exists but lacks a tool the command block pipes to; the task then dies with `command not found`. Single-tool biocontainer images are a frequent source: an image named for one tool often does NOT carry the others in the same pipeline. +6. IF a task needs several tools in one command, use a multi-tool image (for example a `mulled-v2-*` biocontainer). This changes the repository, so treat it as URI replacement per step 3, and record every tool's version in `container_inventory.csv` — they MAY differ from the single-tool images they replace. +7. Create `healthomics.inputs.json` with an ECR registry base path parameter IF container references are parameterized. **Done WHEN**: -- `container_inventory.csv` documents all containers. -- Migration script pushes all containers to ECR. -- All WDL task runtime sections use ECR URIs. -- Zero references to external registries remain. -- At least 5 key containers verified accessible from ECR. +- `container_inventory.csv` documents all containers and records the approach chosen for each. +- All containers are pullable by HealthOmics from private ECR. +- IF using URI replacement: all WDL task runtime sections use ECR URIs, and zero references to external registries remain. +- IF using a registry map: every external registry still referenced in the WDL has a mapping entry, and the map is passed to `CreateAHOWorkflow`. +- For every task, each command invoked in its `command` block is confirmed present in that task's image using the check in step 5. ### Phase 2: Runtime Attribute Audit @@ -193,14 +206,39 @@ AWS HealthOmics requires: ## Technical Patterns +### Registry Map or URI Replacement + +Decide per container by asking whether the migration changes the image's TRANSPORT or its IDENTITY. + +| The change is | Same repository and tag, served from private ECR | A different repository (different tool set or versions) | +|---|---|---| +| Approach | Registry map | Edit the WDL | +| WDL | Unmodified — keeps the public URI | `docker`/`container` value replaced | +| Why | The image is identical, so the redirect hides nothing | The command block's correctness now depends on which image is used | + +DO NOT hide an identity change inside a registry map. A task whose runtime says `bwa` while the map redirects it to a different repository executes something other than what the WDL says, and the next reader has no way to see it from the workflow definition. + +PREFER pinning by digest (`repository@sha256:...`) over a mutable tag on either path. A tag can be reassigned upstream, in which case the same WDL resolves to a different image on a later run. + ### Container Runtime (Before/After) + +Both paths add the required `cpu` and `memory` attributes. They differ only in whether the image reference changes. + ```wdl # Before runtime { docker: "quay.io/biocontainers/bwa:0.7.17--h5bf99c6_8" } -# After +# After — registry map path: the image reference is UNCHANGED. +# The map is passed to CreateAHOWorkflow, which redirects quay.io to your pull-through cache. +runtime { + docker: "quay.io/biocontainers/bwa:0.7.17--h5bf99c6_8" + cpu: 4 + memory: "8 GB" +} + +# After — URI replacement path, for when the repository itself changes. runtime { docker: ".dkr.ecr..amazonaws.com/workflow-name/bwa:0.7.17--h5bf99c6_8" cpu: 4 @@ -252,7 +290,7 @@ workflow MyWorkflow { - S3 bucket(s) created with appropriate permissions - HealthOmics service access - HealthOmics MCP server -- Docker/Finch/Podman installed for container operations +- Docker/Finch/Podman installed for container operations, including verifying image contents (Phase 1) ## References