feat: Checkpoint Requests

[stevensJourney]

Overview

This adds the PowerSync service component for requestCheckpoint, as mentioned in these proposals:

• https://github.com/orgs/powersync-ja/discussions/324
• https://github.com/orgs/powersync-ja/discussions/317

This is related to the following open PRs:

• feat: checkpoint requests powersync-sqlite-core#198

This PR adds a /sync/checkpoint-request route which clients can use to create Checkpoint requests.

Checkpoint requests currently flow through the standard write checkpoint flow in the sync protocol. The implementation here uses the existing collections/tables for write checkpoints for general checkpoint requests.

Collections

Using the same collections for the previous and current checkpoint requests has a few advantages.

Sync protocol

Checkpoint requests currently flow through the existing write_checkpoint marker in Checkpoint started events. We use the existing lastWriteCheckpoint logic for this. This works regardless of the checkpointing method used by the client.

Migrations

Client and PowerSync service versioning migrations (upgrades and downgrades) are almost compatible by default (Bucket storage migrations are required for MongoDB indexes and for new Postgres storage columns). If an existing client has a current write checkpoint record - the client_id is preserved - future requests are monotonically increasing IDs (there are some exceptions to this though - more on that later).

Cleanup

One of the Current Issues in https://github.com/orgs/powersync-ja/discussions/317 are

If an app has many anonymous/temporary users, or regularly creates new temporary databases with unique client ids, it may end up with many write checkpoints on the service. We can never clean these up, since we don't know whether a client would ever connect again and need that write checkpoint. While each individual write checkpoint is slow, this can add up over time when you have hundreds of thousands of unique users/clients.

The goal is to have requested checkpoint records be temporary, where records can be deleted after a period of time. For the current write checkpoint requests, we can never clean these up. Using the same collection allows us to update/mark existing write checkpoints as requested allowing these to be deleted.

Update: This PR handles cleanup by marking checkpoint records correlating to client checkpoint requests with a checkpoint_requested_at Date field. The existing compact job will now delete records where the checkpoint_requeested_at < (now - 30 days). The 30 day limit is configurable via the config.api_parameters.checkpoint_request_retention_days parameter (only configurable for self hosted).

Details

The Postgres and MongoDB bucket storage implementations have been updated to accommodate the current - auto incrementing write-checkpoint2.json endpoint behaviour or the new ability to specify a requested checkpoint ID.

The storage update behaviour diverges based off id a requested checkpoint ID has been supplied or not.

No-ops

https://github.com/orgs/powersync-ja/discussions/317 mentions that checkpoint requests should be no-ops when the request_id is unchanged. This PR takes this slightly further and also asserts the requested checkpoint ID should be larger than the currently stored value. The PowerSync service will return the larger value as part of the sync/checkpoint-request response. Clients can use this information to correct for certain edgecases.

No-ops in this case also prevent the advancing of the replication head if no changes were made in a checkpoint batch. For write-checkpoint2.json requests: we always will make a change and will always advance the replication head. For requested checkpoints, if we received only duplicate requests - we attempt to skip the emission of a replication event. More details of this are mentioned in code comments.

Client Migrations

The client needs to track and manage an increasing checkpoint_request_id. For existing users, this means they might have an existing write checkpoint record. The client should start its request sequence at or above this value (if it exists) in order to prevent setting a target_op below a consistency boundary.

Clients typically also re-issue checkpoint requests on connect (due to their temporary nature). If a newly migrated client does not have the checkpoint request id sequence seeded, it is free to attempt a checkpoint request at 1. If a record exists, the service will reject this ID and return the current largest request ID - which the client can detect and re-seed its sequence.

Note: This has one large caveat if we delete checkpoint request records. If a client does not have a seeded checkpoint request value and the service deleted the record - the client would have to start from 1. This could be acceptable due to the following:

• If the client was using the older write checkpoint method and just migrated:
  • If it was pending a target op, the write checkpoint record should still be present in the DB and can be seeded back to the client
• If the client was not pending any target (which should be the case after a disconnectAndClear), the next write checkpoint could theoretically start at 1 and resolve correctly (I believe)

One important factor to consider here depends on how we track the checkpoint_request_id on the client. If we store a single value per session (clear it in disconnectAndClear), we have the potential to reset the sequence very often if the service record has been deleted.

We could theoretically attempt to store a persisted table of sequences for the user_id/client_id - that would require extracting the user_id somehow which could be more complicated.

As an update: We won't store a table of user sequences, we rather now accept that it's fine for the checkpoint request ID sequence to reset (to 1) if both the client and service have no last known state.

Custom Write Checkpoints

For custom checkpoint requests (previously custom write checkpoints). The general flow is:

• The client generates the next checkpoint_request_id
• The SDK calls a BackendConnector method to post the request ID to the application's backend.
• The backend should match the behaviour seen in our sync/checkpoint-request route - only increment it's write_checkpoint/checkpoint_request record if the supplied ID is larger than the stored value and always report the stored value (used for hydration in the SDK).

The core storage has been updated to accept a checkpoint_requested_at optional Date value. The presence of this value will indicate that the custom write checkpoint record stems from a custom checkpoint request. This means we're free to delete this record in the storage. The external module, which parses incoming sync events and calls the storage has the freedom to decide how/when to pass this parameter to the storage. The specifics for this are not yet implemented, but, we could do either

Add a new column to indicate checkpoint-request nature (untested)

event_definitions:
  # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans.
  write_checkpoints:
    payloads:
      #  This defines where the replicated custom Write Checkpoints should be extracted from
-      - SELECT user_id, checkpoint, client_id FROM checkpoints
+      - SELECT user_id, checkpoint, client_id, true as is_checkpoint_request FROM checkpoints

OR, use a new event definition name.

event_definitions:
  # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans.
-  write_checkpoints:
+ checkpoint_requests:
    payloads:
      #  This defines where the replicated custom Write Checkpoints should be extracted from
      - SELECT user_id, checkpoint, client_id FROM checkpoints

---

AI Disclosure: The following was implemented by first doing a basic implementation by hand - then various changes were assisted by Claude Opus and Codex 5.5.

[changeset-bot]

🦋 Changeset detected

Latest commit: 9115ca6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 19 packages

Name	Type
@powersync/lib-services-framework	Minor
@powersync/service-core	Minor
@powersync/service-core-tests	Minor
@powersync/service-module-convex	Minor
@powersync/service-module-mongodb	Minor
@powersync/service-module-mongodb-storage	Minor
@powersync/service-module-mssql	Minor
@powersync/service-module-mysql	Minor
@powersync/service-module-postgres	Minor
@powersync/service-module-postgres-storage	Minor
@powersync/service-types	Minor
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
@powersync/service-image	Minor
test-client	Patch
@powersync/service-schema	Minor
@powersync/service-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

Looks good so far!

Some minor issues picked up by Codex listed below. I didn't double-check these, but these seem like plausible issues at a glance.

1. checkpoint_request_id accepts negative and unbounded values

The route accepts codecs.bigint directly at packages/service-core/src/routes/endpoints/ checkpointing.ts:13, and the codec accepts any signed integer string at libs/lib-services/src/codec/
codecs.ts:53. A first request with -1 can create a negative managed write checkpoint, and a huge
value passes route validation but fails later in Postgres at the ::int8 cast in modules/module-
postgres-storage/src/storage/checkpoints/PostgresWriteCheckpointAPI.ts:109. This should be validated
at the API boundary as a positive int64-compatible checkpoint id.

2. Performance: Postgres turns every supplied request into a source marker, including already-processed duplicates

modules/module-postgres-storage/src/storage/checkpoints/PostgresWriteCheckpointAPI.ts:170
intentionally sets shouldAdvance true for every supplied checkpoint because Postgres storage does not
track processed state. That means stale or duplicate /sync/checkpoint-request calls still execute the
keepalive/logical marker path at modules/module-postgres/src/api/PostgresRouteAPIAdapter.ts:253. This
undercuts the PR’s no-op goal and can add avoidable WAL/source writes under reconnect storms or
repeated client retries. Not necessarily a correctness blocker, but it is a real performance
implication to address or explicitly accept.

[stevensJourney]

Looks good so far!

Some minor issues picked up by Codex listed below. I didn't double-check these, but these seem like plausible issues at a glance.

1. checkpoint_request_id accepts negative and unbounded values

The route accepts codecs.bigint directly at packages/service-core/src/routes/endpoints/ checkpointing.ts:13, and the codec accepts any signed integer string at libs/lib-services/src/codec/ codecs.ts:53. A first request with -1 can create a negative managed write checkpoint, and a huge value passes route validation but fails later in Postgres at the ::int8 cast in modules/module- postgres-storage/src/storage/checkpoints/PostgresWriteCheckpointAPI.ts:109. This should be validated at the API boundary as a positive int64-compatible checkpoint id.

I've added validation checks to ensure the supplied checkpoint_request_id is in the safe range.

2. Performance: Postgres turns every supplied request into a source marker, including already-processed duplicates

modules/module-postgres-storage/src/storage/checkpoints/PostgresWriteCheckpointAPI.ts:170 intentionally sets shouldAdvance true for every supplied checkpoint because Postgres storage does not track processed state. That means stale or duplicate /sync/checkpoint-request calls still execute the keepalive/logical marker path at modules/module-postgres/src/api/PostgresRouteAPIAdapter.ts:253. This undercuts the PR’s no-op goal and can add avoidable WAL/source writes under reconnect storms or repeated client retries. Not necessarily a correctness blocker, but it is a real performance implication to address or explicitly accept.

The Postgres bucket storage doesn't currently track the processed_at_lsn, which makes performing the optimisation for shouldAdvance difficult. This optimisation is currently disabled for Postgres bucket storage. I've added a comment in the code for now.

[stevensJourney]

This was picked up by Codex, and seems to be a legitimate potential issue. We previously didn't have any uniqueness index based off the user_id. I don't think it was intentional to ever have multiple records for this. I believe it wasn't an issue before since we'd just query a write checkpoint based off the corresponding LSN.
In the new model, we'd like to track a fixed reference per user_id.

[rkistner]

This migration looks risky:

1. The aggregation query can be slow and need to use a lot of db memory if there are many existing write checkpoints on the instance.
2. In theory, new duplicates can be introduced between performing the de-duplication and creating the unique index.
3. There is a period between dropping the old index and creating the new index, in which checkpoints queries will not have any per-user index to use.

In practice it could be okay:

1. I'm not aware of any current case of more than 100k or so write checkpoints in a single instance, but I don't have any data on self-hosted instances.
2. In practice, duplicates should be very rare, so new ones should not be introduced in this period.
3. You can work around this by using a different name for the new index, and creating the new index before dropping the old one. A caveat is that it needs an unique set of fields for each index - you can't create a second one on just {user_id: 1}.

[rkistner]

I'm happy with this for the most part - the only potential issue I could see is the migration for unique user_id.

I'm wondering, what is the effect if there are duplicates? Is it more significant from before this PR?

In practice, since we always use findOneAndUpdate on a user_id, duplicates should be very rare. So I'm wondering if we should do that migration as a separate project, potentially as part of a storage version change.

[rkistner]

This migration looks risky:

1. The aggregation query can be slow and need to use a lot of db memory if there are many existing write checkpoints on the instance.
2. In theory, new duplicates can be introduced between performing the de-duplication and creating the unique index.
3. There is a period between dropping the old index and creating the new index, in which checkpoints queries will not have any per-user index to use.

In practice it could be okay:

1. I'm not aware of any current case of more than 100k or so write checkpoints in a single instance, but I don't have any data on self-hosted instances.
2. In practice, duplicates should be very rare, so new ones should not be introduced in this period.
3. You can work around this by using a different name for the new index, and creating the new index before dropping the old one. A caveat is that it needs an unique set of fields for each index - you can't create a second one on just {user_id: 1}.

[rkistner]

Instead of a normal index, you can create a TTL index, that will automatically remove them. That would avoid the need for having a separate compact task. But that may make it more difficult to keep the retention period configurable, so not sure if this is worth it.

[rkistner]

I wonder if days is the best unit here - I imagine we could remove records in as little as 15-60 minutes?

[stevensJourney]

The thought process here was that it depends on the replication lag. For the write-checkpoint case, we want to make sure the server actually sends the corresponding write-checkpoint back which the client is waiting for.

For a basic case: if the lag is larger than the retention period, we might miss those events.

I guess this could be solved if the client was constantly retrying the last checkpoint request though. I was wondering what a good reason would be for retrying those requests, this might be one of them.

[stevensJourney]

I'm wondering, what is the effect if there are duplicates? Is it more significant from before this PR?

I think this was less of an issue for the old generated write checkpoint flow. Even if duplicate rows existed, the client used the client_id returned from the row that was just updated, so as long as that row had the correct lsns, the acknowledgement could still work.

For the new client-supplied flow, duplicates are more awkward because storage needs to answer “what is the latest monotonic request id for this full user id?” If we query/seed from the wrong duplicate row, we could return a lower checkpoint request id than another row already has, which would put the client in a worse state.
In practice, duplicate creation should still be unlikely because SDKs don’t make concurrent initial requests for the same full user id: they make one initial request, then later requests only after initialization/state is established. The unique index is mainly a guardrail to enforce the one-row-per-full-user-id invariant.

In practice, since we always use findOneAndUpdate on a user_id, duplicates should be very rare. So I'm wondering if we should do that migration as a separate project, potentially as part of a storage version change.

I think there is a very slight risk that duplicate records could exist, but, I think it's so theoretical - that we probably could push this migration out for later.

[stevensJourney]

The change here should cater for the very rare possibility that we might have duplicate write_checkpoint records for a full user_id.

Instead of doing a migration to deduplicate, this now will handle those cases as part of the sync/checkpoint-request flow.

If the client doesn't have the latest client_id/next_checkpoint_request_id sequence. It will make a checkpoint request with the value of 1. Which will funnel through to this logic which will attempt to update all records for the user_id - skipping updates if the currently stored value is larger than the supplied value.

We then query the write checkpoint records for the user and return the MAX client_id for that user (taking all potential duplicates into account).

The client is then free to start its next_checkpoint_request_id at that MAX value. Which will prevent any missed updates in future.

The client's next request will then update all the user's write_checkpoint records, which will mark them as requested - which means we could automatically clean these up at a later stage.