feat(worker): stream through API Gateway REST V1, and grant kv access in CloudFormation

[ItamarZand88]

Summary

Three fixes to how AWS workers are exposed and permissioned. A worker can now stream its response through an API Gateway REST (V1) API with a custom domain; the CloudFormation package finally grants a linked kv store's data-plane access; and the external AI binding gets a service tag that doesn't collide with the external Postgres binding.

When a streaming worker is provisioned:

1. The controller reads WORKER_RESPONSE_STREAMING=true from the worker's environment and takes the REST V1 path instead of the V2 HTTP API.
2. It creates the REST API, a {proxy+} resource, an ANY method, and an AWS_PROXY integration — the integration is the heart: it targets the Lambda's response-streaming-invocations URI with responseTransferMode = STREAM and a 900s timeout, which is what lets bytes reach the client as they are produced.
3. It deploys the prod stage, tags it, and attaches the custom domain through a base path mapping.
4. Buffered workers are untouched: no flag, no REST API, same V2 path as before.

A Lambda Function URL also streams, but cannot carry a custom domain, so REST V1 is the only flavor that does both.

What was broken

• A packaged CloudFormation install could not use its kv store. Linking a kv to a permission profile emitted no IAM at all, so a Worker got no DynamoDB data-plane access — while the identical stack worked on push, where the runtime permissions generator supplies the grants. Queue, storage, vault, ai, email and OpenSearch already emitted these policies; kv was the only data resource that did not.
• A workload could not link both an ambient AI resource and a BYO Postgres. Both bindings serialized "service": "external", and service tags must be globally unique because serde dispatches on the tag alone. The gateway launcher claims every ALIEN_*_BINDING whose tag is in its AI set and fails fast when one doesn't parse as an AI binding, so such a workload died at startup with missing field 'provider'.
• A worker could not stream. The V2 HTTP API buffers, capping responses at 29 seconds and delivering SSE in one lump.

What I did

Added the REST V1 client and controller states; added kv IAM emission mirroring the queue emitter; renamed the external AI binding's tag to external-ai. The permission-owner scan was byte-identical in the queue and email emitters, so it moved to helpers.rs rather than gaining a third copy.

Files touched

crates/alien-aws-clients/src/aws/apigateway.rs (new REST V1 client) · crates/alien-infra/src/worker/aws.rs (controller states) · crates/alien-cloudformation/src/emitters/aws/{kv,queue,email,helpers}.rs · crates/alien-core/src/bindings/ai.rs · crates/alien-ai-gateway/src/config.rs · crates/alien-permissions/permission-sets/worker/{management,provision}.jsonc · packages/ai-gateway/src/binding.ts · regenerated schemas and snapshots.

How I tested

Deployed a streaming worker to a real AWS account and drove it end to end:

• aws apigateway get-integration reports AWS_PROXY, responseTransferMode=STREAM, timeoutInMillis=900000, and the response-streaming-invocations URI.
• SSE arrives incrementally: 31 deltas across 7 distinct 100ms windows. On the buffered path every delta lands in one window.
• A long generation streamed 3,120 deltas with the last at 73.7 seconds and a clean message_stop — well past the 29s cap the V2 API imposes.
• A second install with the AI gate off carries no AI binding and no Bedrock statements on its role, and an invalid key gets the upstream provider's own authentication error back, proving passthrough.
• The kv fix has a test that fails against main and passes here. I also rendered a template on main with a wildcard kv/* grant to check whether the service-account inline policy covered it — it does not; main emits no DynamoDB data-plane action anywhere in CloudFormation.
• The tag collision has a regression test whose failing run reproduces the exact startup crash.
• cargo test: cloudformation 87, core 479, permissions 130, aws-clients 43, infra 518. Clippy clean.

Reviewed for the security questions this touches: the new /restapis and basepathmappings grants mirror the existing /apis and apimappings entries in action set, ARN shape and tag condition — no statement was added without one; the stage is now tagged so it sits inside the same boundary every management grant is conditioned on (I confirmed on a live deployment that it previously was not); the kv policy is pinned to the table's ARN via Fn::GetAtt, so it cannot reach another table in the account; and no grant reaches /apikeys or execute-api:Invoke. Nothing turned up.

Known limitations

• The endpoint flavor is chosen when the endpoint is created. Enabling streaming on a worker that already has a V2 API applies the environment change but does not migrate the endpoint, so that worker needs a fresh install. Migration means moving a custom domain between APIs and belongs in its own change.
• The importer accepts the REST fields but nothing emits them yet, so an imported streaming worker still rebuilds its endpoint state from the heartbeat.
• create_resource has no conflict read-back: a lost response on that one call needs a delete rather than a retry. Same shape as the existing V2 domain path.
• Terraform still emits no resource-scoped kv grants on any cloud. Pre-existing, and unchanged here — this narrows the CloudFormation half only.
• The four "ID not returned" checks use CloudPlatformError with no source, matching the V2 code beside them; switching them all to a concrete variant is a separate cleanup.

[greptile-apps]

Greptile Summary

This follow-up scopes Lambda invocation permission to the selected API Gateway and makes REST stage tagging retryable after partial provisioning failures.

• Adds an API-specific SourceArn to the shared REST V1 and HTTP API V2 Lambda permission.
• Retries REST stage tagging independently after the deployment identifier has been recorded.
• Introduces REST V1 response-streaming infrastructure, CloudFormation KV data-plane grants, and a distinct external AI binding tag.

Confidence Score: 5/5

The PR appears safe to merge because both previously reported failures are fixed and no eligible blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/alien-infra/src/worker/aws.rs	The prior invocation-scope and stage-tag retry defects are both addressed without an eligible distinct residual failure.
crates/alien-aws-clients/src/aws/apigateway.rs	Adds the signed REST V1 API Gateway operations and request models needed for streaming worker endpoints.
crates/alien-cloudformation/src/emitters/aws/kv.rs	Adds resource-scoped DynamoDB data-plane policy emission for linked KV permission profiles.
crates/alien-core/src/bindings/ai.rs	Separates the external AI serialization tag from other external resource bindings.

Sequence Diagram

sequenceDiagram
  participant Controller as AWS Worker Controller
  participant REST as API Gateway REST V1
  participant Lambda as Lambda
  Controller->>REST: Create deployment and prod stage
  REST-->>Controller: Deployment ID
  Controller->>REST: Tag prod stage
  alt Tagging fails transiently
    REST-->>Controller: Error
    Controller->>REST: Retry stage tagging
  end
  Controller->>Lambda: Add invoke permission with API-specific SourceArn
  Lambda-->>Controller: Permission recorded

Loading

_{Reviews (2): Last reviewed commit: "fix(worker): scope the API Gateway invok..." | Re-trigger Greptile}