feat(backend): background GC jobs, schema audit, device capability negotiation, ciphertext-only migration

apps/backend/docs/message-encryption-migration.md

@@ -0,0 +1,135 @@
+# Message encryption migration: plaintext → ciphertext-only
+
+This document is the canonical reference linked from `GET /conversations/:id/search`'s
+410 response and from the schema-overhaul audit — it records the decision behind
+`drizzle/0003_ciphertext_only_messages.sql` and the policy for any plaintext that
+existed before it.
+
+## Background
+
+Earlier in this project's history, `messages` carried a plaintext `content` column
+(plus a GIN index supporting server-side search) alongside the work to move to
+per-recipient E2EE envelopes (`message_envelopes`) and a `ciphertext` column on
+`messages` itself. `docs/e2ee-onboarding.md` documents the current, implemented
+E2EE data model; this document covers the one-time cutover away from the old
+plaintext column and what happens to data that predates it.
+
+Server-side message search was removed as part of this cutover — the server
+cannot search ciphertext it cannot read, so `GET /conversations/:id/search` now
+returns `410 Gone` pointing here, and search is client-side over decrypted
+messages.
+
+## Decision: archive-then-purge (not a tombstone)
+
+Two options were considered for handling rows where `messages.content` is
+non-null at migration time:
+
+1. **Tombstone in place** — keep a `content`-shaped column (or a
+   `legacyPlaintext` column) on `messages` forever, nulled out or flagged, so
+   the "this row used to be plaintext" fact stays attached to the row.
+2. **Archive then purge** (chosen) — copy every non-null `content` value into
+   a separate `message_content_archive` table, then drop the column from
+   `messages` entirely.
+
+**Archive-then-purge was chosen** because:
+
+- It keeps `messages` fully ciphertext-shaped going forward. Every code path
+  that reads a message (`serializeMessage()` in `lib/messages.ts`, the
+  `conversations`/`messages` routes, the socket layer) only ever has a
+  `ciphertext` branch to reason about — there is no lingering plaintext
+  branch that a future change could accidentally serve to a client as if it
+  were normal ciphertext.
+- `message_content_archive` has **no route or query path through the app's
+  API** (see `db/schema.ts` — the table is defined but never imported by
+  anything in `routes/`, `services/`, or `socket/`). It exists purely for
+  legal/compliance/audit access on its own retention schedule, separate from
+  the live message store's.
+- It avoids growing the "hot path" `messages` table with a rarely-needed
+  legacy column and its GIN index indefinitely.
+
+The rejected tombstone approach was ruled out because it permanently taxes
+every future `messages` migration and query with a legacy-shaped column for
+data that, by definition, existed only before this cutover and will never
+grow again.
+
+## No silent E2EE claim over old plaintext
+
+Rows whose plaintext was archived are **not** presented to clients as if they
+were ciphertext. Once `content` is dropped, `serializeMessage()`'s existing
+fallback chain (envelope → base `ciphertext` → `unavailable: true`) means a
+pre-cutover row with no `ciphertext` and no matching envelope now correctly
+resolves to `{ ciphertext: null, unavailable: true }` — the client sees the
+message as unavailable rather than being shown a decrypted-looking payload
+that was, in fact, sent before E2EE existed. Nothing in this migration
+retroactively re-encrypts old plaintext or backdates an "encrypted" claim
+onto it.
+
+## What the forward migration does
+
+`drizzle/0003_ciphertext_only_messages.sql`:
+
+1. Creates `message_content_archive` (id, `original_message_id` — intentionally
+   **not** a foreign key, so hard-deleting a `messages` row never cascades
+   into the archive — `conversation_id`, `sender_id`, `content`,
+   `original_created_at`, `archived_at`).
+2. If (and only if) `messages.content` still exists — checked dynamically via
+   `information_schema.columns`, since a plain SQL statement referencing a
+   column that doesn't exist fails to parse — copies every non-null row into
+   the archive.
+3. Drops the column's GIN index. The exact index name from before this
+   repo's migration history was squashed to a single `0000` migration is not
+   recoverable, so several plausible historical names are dropped
+   defensively with `DROP INDEX IF EXISTS` (a no-op for any name that
+   doesn't match).
+4. Drops `messages.content` with `DROP COLUMN IF EXISTS` — a no-op on a
+   database that never had it, which is the case for this repo's own `0000`
+   migration.
+5. Defensively ensures the ciphertext-model columns/tables (`messages.ciphertext`,
+   `senderDeviceId`, `fileId`, `editsMessageId`, `deletedAt`,
+   `message_envelopes`) exist, for any database whose applied-migration
+   history predates the `0000` squash. On every database that already ran
+   `0000` this is a pure no-op.
+
+This makes the migration safe to run both against this repo's current
+(already-ciphertext) schema and against a hypothetical legacy database that
+still has the old plaintext column and a populated `messages` table.
+
+## Rollback plan
+
+`drizzle-kit` has no built-in down-migration runner, so the rollback lives at
+`drizzle/rollback/0003_ciphertext_only_messages.down.sql` and is invoked
+manually, e.g.:
+
+```sh
+psql "$DATABASE_URL" -f apps/backend/drizzle/rollback/0003_ciphertext_only_messages.down.sql
+```
+
+It:
+
+1. Re-adds `messages.content` (nullable).
+2. Recreates a GIN full-text index over it.
+3. Restores plaintext from `message_content_archive` back into `messages.content`
+   for every row the archive has a match for.
+
+**Rollback limitations** (also called out at the top of the script itself):
+
+- Only rows present in the archive get plaintext restored. Any message sent
+  *after* the forward migration ran was only ever stored as ciphertext —
+  there is nothing to restore for it, by design.
+- The recreated index is a reasonable equivalent, not necessarily
+  byte-identical to whatever definition existed pre-squash.
+- The rollback does **not** remove the ciphertext/envelope/device-capability/
+  GC schema this migration set introduces — undoing those would remove the
+  entire E2EE data model, which is a separate, deliberate decision this
+  script does not make on an operator's behalf.
+- `message_content_archive` is left in place after a rollback (nothing is
+  destroyed); drop it manually once you're satisfied the restore is correct.
+
+## Retention of the archive
+
+`message_content_archive` is not currently covered by the background GC jobs
+in `services/deviceGc.ts` / `services/envelopeGc.ts` / `services/fileCleanup.ts`
+— it is intentionally kept until a compliance/legal retention policy for
+pre-E2EE history is decided separately. A future GC pass following the same
+idempotent, env-configurable pattern as the existing jobs would be the
+natural way to enforce that retention once a window is chosen.

apps/backend/drizzle/0001_gc_background_jobs.sql

@@ -0,0 +1,21 @@
+-- Background GC jobs (prekey/envelope/file/device cleanup) — schema support.
+--
+-- Adds:
+--   * mls_key_packages — one-time MLS KeyPackages per device, mirroring
+--     device_prekeys' consumed-flag model so issuance stays auditable.
+--   * devices.stale_flagged_at — informational marker set by the device-GC
+--     job once a revoked device ages past retention. Never deletes the row.
+CREATE TABLE "mls_key_packages" (
+	"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
+	"device_id" uuid NOT NULL,
+	"key_package" text NOT NULL,
+	"consumed" boolean DEFAULT false NOT NULL,
+	"consumed_at" timestamp,
+	"created_at" timestamp DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+ALTER TABLE "mls_key_packages" ADD CONSTRAINT "mls_key_packages_device_id_devices_id_fk" FOREIGN KEY ("device_id") REFERENCES "public"."devices"("id") ON DELETE cascade ON UPDATE no action;
+--> statement-breakpoint
+CREATE INDEX "mls_key_packages_device_available_idx" ON "mls_key_packages" USING btree ("device_id") WHERE "mls_key_packages"."consumed" = false;
+--> statement-breakpoint
+ALTER TABLE "devices" ADD COLUMN IF NOT EXISTS "stale_flagged_at" timestamp;

apps/backend/drizzle/0002_device_capabilities.sql

@@ -0,0 +1,8 @@
+-- Device capability/version negotiation (#180-follow-on).
+--
+-- Advertises supported protocols/ciphersuites/file-transfer versions per
+-- device so senders can pick an encryption path both sides support. Rows
+-- written before this migration default to the sealed_box-only baseline —
+-- the protocol every device in this codebase already implements — so
+-- existing devices negotiate correctly without any backfill.
+ALTER TABLE "devices" ADD COLUMN IF NOT EXISTS "capabilities" jsonb DEFAULT '{"protocols":["sealed_box"],"ciphersuites":[],"fileTransfer":[]}'::jsonb NOT NULL;

apps/backend/drizzle/0003_ciphertext_only_messages.sql

@@ -0,0 +1,102 @@
+-- One-time migration: drop plaintext `messages.content`, reset to the
+-- ciphertext-only model.
+--
+-- Policy (documented in full at docs/message-encryption-migration.md):
+-- ARCHIVE THEN PURGE. Any existing plaintext row is copied into
+-- `message_content_archive` — a table with no read path through the app's
+-- API — before the column is dropped from `messages`. This keeps `messages`
+-- fully ciphertext-shaped going forward (so `serializeMessage()` never has a
+-- plaintext branch to accidentally serve) while preserving a compliance/
+-- audit copy of pre-E2EE history on its own retention schedule, instead of
+-- silently destroying it or leaving a tombstone column on `messages` forever.
+--
+-- Safe to run on a DB that never had a `content` column (this repo's own
+-- migration history: 0000 already created `messages` in its ciphertext
+-- shape) — the archive step is a no-op guarded by an information_schema
+-- check, and every DDL statement below uses IF EXISTS/IF NOT EXISTS so
+-- nothing errors either way. See docs/message-encryption-migration.md for
+-- the full rollback plan and its limitations.
+
+CREATE TABLE IF NOT EXISTS "message_content_archive" (
+	"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
+	-- Intentionally not a foreign key: this archive must outlive the
+	-- `messages` row it was copied from (e.g. hard-deletion by the message
+	-- GC path must not cascade-delete the compliance copy).
+	"original_message_id" uuid NOT NULL,
+	"conversation_id" uuid,
+	"sender_id" uuid,
+	"content" text NOT NULL,
+	"original_created_at" timestamp,
+	"archived_at" timestamp DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE INDEX IF NOT EXISTS "message_content_archive_original_message_idx" ON "message_content_archive" USING btree ("original_message_id");
+--> statement-breakpoint
+
+-- Archive any existing plaintext before the column is dropped. Dynamic SQL
+-- is required here (unlike the DROP statements below) because a plain
+-- top-level statement referencing `messages.content` would fail to parse on
+-- a database where that column never existed.
+DO $$
+BEGIN
+  IF EXISTS (
+    SELECT 1 FROM information_schema.columns
+    WHERE table_schema = 'public' AND table_name = 'messages' AND column_name = 'content'
+  ) THEN
+    EXECUTE '
+      INSERT INTO message_content_archive
+        (original_message_id, conversation_id, sender_id, content, original_created_at)
+      SELECT id, conversation_id, sender_id, content, created_at
+      FROM messages
+      WHERE content IS NOT NULL
+    ';
+  END IF;
+END $$;
+--> statement-breakpoint
+
+-- Drop the plaintext column's GIN index. The exact index name from
+-- pre-squash history is not recoverable, so every plausible historical name
+-- is dropped defensively — DROP INDEX IF EXISTS is a no-op for names that
+-- don't exist, so this is safe regardless of which (if any) actually match.
+DROP INDEX IF EXISTS "messages_content_gin_idx";
+--> statement-breakpoint
+DROP INDEX IF EXISTS "messages_content_search_idx";
+--> statement-breakpoint
+DROP INDEX IF EXISTS "messages_content_tsv_idx";
+--> statement-breakpoint
+DROP INDEX IF EXISTS "messages_content_idx";
+--> statement-breakpoint
+
+ALTER TABLE "messages" DROP COLUMN IF EXISTS "content";
+--> statement-breakpoint
+
+-- Defensively ensure the ciphertext-model columns/tables this migration's
+-- companion schema overhaul introduced are present, for any database whose
+-- migration history predates the 0000 squash and therefore skipped them.
+-- A no-op everywhere this repo's own 0000 migration already ran.
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "ciphertext" text;
+--> statement-breakpoint
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "sender_device_id" uuid;
+--> statement-breakpoint
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "file_id" uuid;
+--> statement-breakpoint
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "edits_message_id" uuid;
+--> statement-breakpoint
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "deleted_at" timestamp;
+--> statement-breakpoint
+
+-- Note: this fallback intentionally omits FK constraints — on every database
+-- that already ran this repo's 0000 migration (the only realistic case),
+-- the table already exists with its constraints and this is a pure no-op.
+-- A database old enough to hit this branch predates the squash entirely and
+-- needs a manually-reviewed reconciliation, not a silent constraint add.
+CREATE TABLE IF NOT EXISTS "message_envelopes" (
+	"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
+	"message_id" uuid NOT NULL,
+	"recipient_device_id" uuid NOT NULL,
+	"recipient_user_id" uuid NOT NULL,
+	"ciphertext" text NOT NULL,
+	"delivered_at" timestamp,
+	"read_at" timestamp,
+	"created_at" timestamp DEFAULT now() NOT NULL
+);

apps/backend/drizzle/rollback/0003_ciphertext_only_messages.down.sql

@@ -0,0 +1,35 @@
+-- Rollback for 0003_ciphertext_only_messages.sql.
+--
+-- drizzle-kit has no built-in "down" migration runner — this script is
+-- invoked manually (e.g. `psql "$DATABASE_URL" -f drizzle/rollback/0003_ciphertext_only_messages.down.sql`)
+-- and is NOT part of the drizzle migration journal, so it is never applied
+-- automatically by `db:migrate`.
+--
+-- LIMITATIONS (see docs/message-encryption-migration.md for full detail):
+--   * Only rows present in `message_content_archive` get their plaintext
+--     restored. Any message sent after the forward migration ran was only
+--     ever stored as ciphertext — there is no plaintext to bring back for it.
+--   * The recreated GIN index is a reasonable equivalent (full-text search
+--     over `content`), not necessarily byte-identical to whatever indexdef
+--     existed pre-squash — that definition was not recoverable (see the
+--     forward migration's comment on why several index names are dropped
+--     defensively rather than one exact name).
+--   * This script does NOT drop the ciphertext/envelope/device-capability/
+--     GC columns and tables added by this migration set. Rolling those back
+--     would remove the entire E2EE data model, not just this one migration's
+--     change — treat that as a separate, deliberate decision, not a side
+--     effect of undoing the content-column drop.
+--   * `message_content_archive` is left in place after restoring so no data
+--     is destroyed by running this script; drop it manually once satisfied.
+
+ALTER TABLE "messages" ADD COLUMN IF NOT EXISTS "content" text;
+--> statement-breakpoint
+
+CREATE INDEX IF NOT EXISTS "messages_content_gin_idx" ON "messages" USING gin (to_tsvector('english', "content"));
+--> statement-breakpoint
+
+UPDATE "messages" m
+SET "content" = a."content"
+FROM "message_content_archive" a
+WHERE a."original_message_id" = m."id"
+  AND m."content" IS NULL;

apps/backend/src/__tests__/devices.test.ts

@@ -182,6 +182,7 @@ describe('GET /devices', () => {
         'platform',
         'lastSeenAt',
         'revokedAt',
+        'capabilities',
         'oneTimePreKeysRemaining',
       ].sort(),
     );

apps/backend/src/db/schema.ts

@@ -10,9 +10,11 @@ import {
   jsonb,
   uniqueIndex,
   check,
+  jsonb,
   type AnyPgColumn,
 } from 'drizzle-orm/pg-core';
 import { relations, sql } from 'drizzle-orm';
+import { DEFAULT_CAPABILITIES, type DeviceCapabilities } from '../lib/capabilities.js';
 
 export const users = pgTable('users', {
   id: uuid('id').primaryKey().defaultRandom(),
@@ -214,6 +216,18 @@ export const devices = pgTable(
     lastSeenAt: timestamp('last_seen_at'),
     pushEnabled: boolean('push_enabled').notNull().default(true),
     revokedAt: timestamp('revoked_at'),
+    // Set by the device-GC job once a revoked device has aged past
+    // DEVICE_STALE_AFTER_DAYS. Purely informational — flags the row for
+    // admin visibility/future hard-delete without destroying audit history.
+    staleFlaggedAt: timestamp('stale_flagged_at'),
+    // Supported protocols/ciphersuites/file-transfer versions this device
+    // advertises (lib/capabilities.ts). Defaults to the sealed_box-only
+    // baseline so rows written before this column existed, or clients that
+    // never send it, negotiate correctly rather than erroring — see
+    // lib/capabilities.ts `normalizeCapabilities`.
+    capabilities: jsonb('capabilities').$type<DeviceCapabilities>().notNull().default(
+      DEFAULT_CAPABILITIES,
+    ),
     createdAt: timestamp('created_at').notNull().defaultNow(),
     updatedAt: timestamp('updated_at').notNull().defaultNow(),
   },
@@ -569,6 +583,7 @@ export const tokenTransfersRelations = relations(tokenTransfers, ({ one }) => ({
 export const devicesRelations = relations(devices, ({ one, many }) => ({
   user: one(users, { fields: [devices.userId], references: [users.id] }),
   prekeys: many(devicePrekeys),
+  mlsKeyPackages: many(mlsKeyPackages),
   messages: many(messages),
   pushSubscriptions: many(pushSubscriptions),
   keyHistory: many(deviceKeyHistory),
@@ -583,6 +598,10 @@ export const devicePrekeysRelations = relations(devicePrekeys, ({ one }) => ({
   device: one(devices, { fields: [devicePrekeys.deviceId], references: [devices.id] }),
 }));
 
+export const mlsKeyPackagesRelations = relations(mlsKeyPackages, ({ one }) => ({
+  device: one(devices, { fields: [mlsKeyPackages.deviceId], references: [devices.id] }),
+}));
+
 export const pushSubscriptionsRelations = relations(pushSubscriptions, ({ one }) => ({
   device: one(devices, { fields: [pushSubscriptions.deviceId], references: [devices.id] }),
 }));
@@ -624,3 +643,5 @@ export type Device = typeof devices.$inferSelect;
 export type NewDevice = typeof devices.$inferInsert;
 export type DevicePrekey = typeof devicePrekeys.$inferSelect;
 export type NewDevicePrekey = typeof devicePrekeys.$inferInsert;
+export type MlsKeyPackage = typeof mlsKeyPackages.$inferSelect;
+export type NewMlsKeyPackage = typeof mlsKeyPackages.$inferInsert;

apps/backend/src/index.ts

@@ -45,6 +45,8 @@ import {
   runForever as runStellarListener,
 } from './services/stellarListener.js';
 import { startFileCleanupJob } from './services/fileCleanup.js';
+import { startDeviceGcJob } from './services/deviceGc.js';
+import { startEnvelopeGcJob } from './services/envelopeGc.js';
 import { loadEnv } from './config.js';
 import { getObjectStore } from './lib/objectStore.js';
 import { presenceChurnTotal, connectedSockets } from './lib/metrics.js';
@@ -427,6 +429,13 @@ void attachRedisAdapter();
 // #231 – start background file cleanup + push backoff re-enable job
 startFileCleanupJob();
 
+// Background GC: prune consumed/expired prekeys + MLS key packages, flag
+// long-revoked devices, and delete fully-delivered/expired message envelopes.
+// Retention windows are configurable via env (see services/deviceGc.ts and
+// services/envelopeGc.ts) so operators can tune them per deployment.
+startDeviceGcJob();
+startEnvelopeGcJob();
+
 // Subscribe to device_revoked:* channels so any gateway instance can
 // disconnect a revoked device's sockets within seconds, even when the
 // revocation was issued on a different node.