fix(infra): make prod RDS reachable from the lambdas - #292
Merged
Conversation
GET /auth/me and every other DB-backed endpoint hung for ~30s and returned nothing. Nothing in the deployed stack could reach the production database. Three independent defects, all of which had to be fixed for a single request to work: 1. aws_db_instance.branch_rds never set publicly_accessible, so it defaulted to false and its endpoint only resolved privately inside the default VPC. The six lambdas have no vpc_config and run outside any VPC, so they had no route to it. Set publicly_accessible = true. The stronger fix is moving the lambdas into the VPC, but a VPC lambda with no internet route cannot reach Cognito -- which all six do on cold start to verify JWTs -- so it requires a NAT gateway (~$33/mo) or a cognito-idp interface endpoint (~$7/mo). Deliberately traded away. 2. rds.force_ssl = 1 on the default.postgres17 parameter group, but all six db.ts hardcoded ssl: false, so Postgres would have rejected the connections even once routable. Enable TLS in production only, since local docker-compose Postgres has no TLS. 3. pg had no connectionTimeoutMillis. A blackholed SYN never gets an RST, so the pool waited forever and turned a network fault into a 30s function timeout with nothing logged -- which is why this presented as a hang rather than an error. Set to 5s, well inside API Gateway's 29s ceiling. Also replaces the VPC default security group on the instance, which allowed all protocols from 0.0.0.0/0, with branch-rds-sg scoped to 5432. This matters more now the endpoint is public. It is still open to the internet, because AWS publishes no Lambda egress ranges (no LAMBDA tag in ip-ranges.json; non-VPC lambdas egress from the region's ~4.4M-address EC2 pool), so no narrower CIDR would admit them. The master password is therefore the real control. Adds prevent_destroy, since skip_final_snapshot = true means any replacement would destroy the database with no backup. Follow-ups, not in this PR: rotate the master password, pin the RDS CA bundle so TLS is authenticated rather than only encrypted, and add a mechanism that applies db_setup.sql to prod (nothing in CI ever has). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Auto-formatted .tf files with terraform fmt - Updated README.md with terraform-docs Co-authored-by: nourshoreibah <nourshoreibah@users.noreply.github.com>
Contributor
Terraform Plan 📖
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /auth/mefrom the frontend hung for ~30s and returned nothing. Root cause was not application logic — nothing in the deployed stack has ever been able to reach the production database. That is also why prod had no schema until it was applied by hand.Three defects, all required for one request to work
1. The lambdas had no route to the database.
aws_db_instance.branch_rdsnever setpublicly_accessible, so it defaulted tofalseand its endpoint only resolved to a private address inside the default VPC. The six lambdas have novpc_configand run outside any VPC. Fixed withpublicly_accessible = true.The stronger fix is putting the lambdas in the VPC. Rejected on cost: a VPC lambda with no internet route can't reach Cognito, which all six do on cold start (
aws-jwt-verifyJWKS fetch), so it needs a NAT gateway ($33/mo) or a$7/mo). Consciously traded away for $0.cognito-idpinterface endpoint (2. TLS was disabled but mandatory.
rds.force_ssl = 1ondefault.postgres17, while all sixdb.tshardcodedssl: false— Postgres would have rejected these connections even once routable, withno pg_hba.conf entry for host ..., no encryption. Enabled in production only; local docker-compose Postgres has no TLS.Worth noting this was masked: manual
psqlworked because psql defaults tosslmode=preferand silently negotiated TLS.3. Failures hung instead of erroring.
pghad noconnectionTimeoutMillis. A blackholed SYN never gets an RST, so the pool waited forever, turning a network fault into a 30s function timeout with nothing logged. Set to 5s, well inside API Gateway's 29s integration ceiling.Security change
Replaces the VPC default security group on the instance — which allowed all protocols from
0.0.0.0/0— withbranch-rds-sgscoped to 5432. This was survivable while the instance was unreachable; it would not be now that the endpoint is public.The rule is still
0.0.0.0/0on 5432, and that is deliberate rather than lazy: AWS publishes no Lambda egress ranges (there is noLAMBDAtag inip-ranges.json), and non-VPC lambdas egress from the region's EC2 pool — 83 CIDR blocks, ~4.4M addresses in us-east-2, i.e. every EC2 instance of every AWS customer in Ohio. No narrower CIDR would still admit our own lambdas. The master password is therefore the real control.Also adds
prevent_destroy:skip_final_snapshot = truemeans any replacement (e.g. an innocuous-lookingdb_subnet_group_nameedit) would silently destroy the database with no backup.Verification
terraform validatepasses;terraform fmt -checkclean.tsc --noEmit) after installing workspace deps and buildingshared/lambda-auth.terraform applyand a lambda deploy, thenGET /auth/meshould return the admin user withisAdmin: true.Follow-ups (not in this PR)
rejectUnauthorized: false). Needs a shared package so the PEM isn't copied six times.db_setup.sqlto prod — nothing in CI ever has (lambda-tests.ymlandregenerate-db-types.yamlboth target localhost).db.tsfiles into ashared/lambda-dbpackage.🤖 Generated with Claude Code