Delta Sync, Checkpoints, and Efficient Data Transfer

Why Delta Sync Matters

Delta sync means transferring only what changed since the last successful synchronization, instead of re-downloading or re-uploading entire datasets. In offline-first apps, delta sync reduces bandwidth, lowers battery usage, speeds up perceived performance, and decreases the chance of conflicts because smaller batches finish faster. The core idea is simple: both client and server maintain enough metadata to identify a shared point in time (or version) and then exchange only the differences after that point.

Delta sync is not a single algorithm; it is a family of techniques. Some systems use monotonically increasing revision numbers, some use timestamps, others use per-record version vectors, and others use content hashes. The common requirement is that both sides can answer: “What has changed since checkpoint X?” and can do so efficiently and correctly even when the client has been offline for a long time.

Full sync vs delta sync

• Full sync: client fetches all records (or all records in a scope) and replaces local state. Simple but expensive and risky on mobile networks.
• Delta sync: client fetches only inserts/updates/deletes since a checkpoint. Requires more metadata and careful handling of deletions, ordering, and compaction.

Core Building Blocks: Deltas, Checkpoints, and Cursors

What is a delta?

A delta is a representation of change. In practice, deltas are usually expressed as one of the following:

• Change events: a stream of events like created, updated, deleted with record identifiers and payloads.
• Patch operations: JSON Patch or similar operations that transform an old document into a new one.
• Row-level upserts: “here is the latest version of record A” plus a version marker; deletions are represented separately.

For mobile apps, row-level upserts plus tombstones for deletions are common because they are easy to apply idempotently and can be stored in a local database without needing the previous version to compute a patch.

What is a checkpoint?

A checkpoint is the client’s durable marker of “I have successfully applied all server changes up to this point.” It must be persisted locally (not just in memory), because the app can be killed mid-sync. Checkpoints are often called cursor, sync token, since, or watermark.

Continue in our app.

• Listen to the audio with the screen off.
• Earn a certificate upon completion.
• Over 5000 courses for you to explore!

Or continue reading below...

Download the app

Good checkpoints have these properties:

• Monotonic: newer checkpoints represent a superset of changes compared to older ones.
• Opaque: the server can encode internal state (like log offsets) without exposing details; the client treats it as an opaque string.
• Stable across pages: when paginating, each page should advance a page cursor while still being tied to a consistent snapshot.
• Durable: stored transactionally with applied changes so the client never advances the checkpoint without applying the corresponding data.

Checkpoint types

• Timestamp-based: updatedAt > lastSyncTime. Easy but risky with clock skew, backdated updates, and non-unique timestamps.
• Revision-based: server assigns a global increasing revision number to each change. Stronger ordering and simpler correctness.
• Log offset / changefeed token: checkpoint is an offset into a change log. Efficient and precise; requires server-side retention.
• Per-collection tokens: separate checkpoints per entity type or per scope (workspace/project) to reduce fan-out and allow partial sync.

Designing a Delta Sync Contract (API Shape)

A delta sync API should allow the client to request changes since a checkpoint and receive a response that includes: (1) changes, (2) a new checkpoint, and (3) pagination info if needed. A typical pattern is:

GET /sync/changes?since={checkpoint}&limit=500

Response example:

{  "changes": [    {"type":"upsert","entity":"task","id":"t1","version":1042,"data":{...}},    {"type":"delete","entity":"task","id":"t9","version":1043}  ],  "nextSince": "opaque-token-1043",  "hasMore": false}

Important details:

• Idempotency: applying the same page twice should not corrupt state. Include a per-record version and use upsert semantics.
• Deletions: must be represented explicitly (tombstones). Without them, the client cannot know what to remove.
• Scoping: include scope parameters (workspaceId, userId) so the server can compute deltas efficiently for that subset.
• Consistency snapshot: for multi-page responses, ensure pages are consistent. Either the server returns a snapshot token that pins the result set, or it guarantees that nextSince only advances after the final page.

Snapshot token vs advancing checkpoint per page

Two safe pagination strategies are common:

• Snapshot token: first response returns snapshotToken and a page cursor. Subsequent pages use the same snapshot token to ensure a stable view. At the end, the server returns nextSince to store as the new checkpoint.
• Monotonic log offset: if the checkpoint is a log offset and the server returns changes strictly in log order, it is safe to advance the checkpoint per page as long as the client applies each page transactionally.

Applying Deltas Safely: Transactional Checkpointing

The most common correctness bug in delta sync is advancing the checkpoint without fully applying the changes. The fix is to treat “apply changes + store new checkpoint” as a single atomic operation in the local database.

Step-by-step: applying a page of server changes

• Step 1: Start a local DB transaction. This ensures either everything is applied or nothing is.
• Step 2: For each change event:
  • If type=upsert: write the record with its server version. If the record exists, compare versions and only overwrite if the incoming version is newer (or if your merge policy says so).
  • If type=delete: write a tombstone (or delete locally) in a way that prevents resurrection by older updates. Often this means storing a deletedVersion or keeping a tombstone row with the version.
• Step 3: Persist the new checkpoint (page cursor or final nextSince) in the same transaction.
• Step 4: Commit. Only after commit should the sync engine consider the page “done.”

Example pseudo-code:

db.transaction(() => {  for (change of response.changes) {    if (change.type === 'upsert') {      db.upsert(change.entity, change.id, change.data, change.version)    } else if (change.type === 'delete') {      db.applyTombstone(change.entity, change.id, change.version)    }  }  db.setSyncCheckpoint(scopeKey, response.nextSince)})

Tombstones and “resurrection” prevention

If you physically delete a row locally, an older upsert arriving later could recreate it. To prevent this, keep a tombstone with a version marker, or keep the deleted row with a deletedAt and deletedVersion. When applying an upsert, check whether a tombstone exists with a newer version; if so, ignore the upsert.

Efficient Data Transfer: Reduce Bytes, Round Trips, and Work

Choose the right payload granularity

Delta sync can still be wasteful if each change includes a full document. Options:

• Full record payloads: simplest; good when records are small and changes are infrequent.
• Partial fields: send only changed fields plus a version. Requires careful merge on the client and a clear rule for missing fields (missing means unchanged, not null).
• Patch format: smallest payload for large documents, but requires the client to have the correct base version and to apply patches reliably.

A pragmatic approach is to send full payloads for small entities and patches for large blobs (notes, rich text, JSON configs), possibly behind a feature flag.

Compression and encoding

• HTTP compression (gzip/br) is usually the biggest win for JSON payloads.
• Binary formats (Protocol Buffers, FlatBuffers) can reduce size and parsing cost, but add schema management overhead.
• Field naming: avoid overly verbose keys in hot paths if you control the protocol; compression helps, but parsing still costs CPU.

Batching and limits

Batching reduces round trips but increases memory and failure blast radius. Use a server-side limit and return hasMore with a cursor. On the client, process pages incrementally and yield to the UI thread between pages if needed.

Conditional requests for large objects

Some data should not be pushed through the changefeed as full content (images, attachments). Instead, sync metadata via deltas and fetch blobs separately using conditional requests:

• Sync attachment metadata: id, contentHash, size, updatedVersion.
• Download blob only if local hash differs.
• Use ETag and If-None-Match to avoid re-downloading.

Server-Side Strategies for Producing Deltas

Change log (append-only) + retention

A robust way to implement delta sync is to write every mutation to an append-only change log with a monotonically increasing sequence number. The delta API then reads from that log starting at the client’s checkpoint. This provides strong ordering and makes pagination straightforward.

Key considerations:

• Retention window: you cannot keep logs forever. If a client’s checkpoint is older than the retention window, the server must require a resync strategy (for example, a snapshot rebuild for that scope).
• Compaction: multiple updates to the same record can be compacted when generating deltas (send only the latest state) to reduce payload size, as long as ordering constraints are respected.
• Fan-out: for multi-tenant scopes, partition logs by tenant/workspace to keep queries efficient.

Updated-at queries (with caveats)

Querying by updatedAt can work for small systems but needs safeguards:

• Use a compound cursor: (updatedAt, id) to break ties when multiple rows share the same timestamp.
• Ensure updatedAt is assigned by the server, not the client.
• Be careful with backfills and migrations that touch many rows; they can create huge deltas.

Hybrid: snapshot + incremental

For clients that are very stale or for first-time sync of a scope, the server can provide a snapshot endpoint that returns a consistent dataset plus a checkpoint representing the snapshot boundary. After that, the client switches to incremental deltas from that checkpoint. This avoids forcing the change log to retain data for extremely long periods.

Checkpoint Management Patterns

Per-scope checkpoints

Instead of one global checkpoint, store checkpoints per scope (for example, per workspace) and sometimes per entity group. Benefits:

• Smaller delta queries and smaller responses.
• Parallelizable sync across scopes.
• Partial availability: one scope can be up to date even if another fails.

Tradeoff: more bookkeeping and more server endpoints or parameters.

Two-phase checkpoints for safer pagination

When using snapshot tokens, treat the final nextSince as the only durable checkpoint. During pagination, store an in-progress cursor separately. If the app crashes mid-way, you can resume the snapshot pagination without corrupting the durable checkpoint.

Example local state:

{  "scope": "workspace:123",  "durableSince": "token-9000",  "inProgress": {    "snapshotToken": "snap-abc",    "pageCursor": "cursor-3"  }}

Conflict-Aware Delta Sync: Versions and Causality

Delta sync often interacts with concurrent edits. Even if conflict resolution is handled elsewhere, delta transfer must carry enough metadata to make correct decisions. At minimum, each record should have a version marker that is comparable (or at least orderable) within the server’s authority.

Practical metadata to include per record

• recordVersion: monotonically increasing per record or globally.
• updatedAt: useful for UI and debugging, but not the primary ordering mechanism.
• deleted flag/version: to represent deletions safely.
• serverAssignedId: for records created offline with temporary IDs, include mapping events (see below).

Handling offline-created records: ID mapping deltas

When a client creates a record offline, it may use a temporary ID. After upload, the server may assign a canonical ID. Delta sync should include an explicit mapping change so other devices and the same device can reconcile references.

Example change event:

{"type":"idmap","entity":"task","tempId":"local-123","id":"t77","version":2001}

Client application steps:

• Update the primary key for the record (or store an alias mapping table).
• Update foreign keys referencing the temp ID.
• Mark the mapping as applied idempotently (reapplying should be safe).

Reducing Server Work and Client Work with Delta Compaction

If a record changes multiple times while the client is offline, sending every intermediate change may be unnecessary. Compaction means collapsing multiple changes into a smaller representation.

Compaction approaches

• Latest-state compaction: for each record, send only the newest upsert or delete after the checkpoint. This is effective for “current state” apps (tasks, contacts) where intermediate states are not needed.
• Event-preserving: keep all events when the app needs history (audit logs, chat messages). In that case, the delta is the event stream itself.

Be explicit about which entities are stateful vs eventful. Mixing them in one endpoint is possible, but the client must know how to apply each category.

Step-by-Step: Implementing a Delta Sync Loop with Checkpoints

This walkthrough focuses on the incremental download side (server to client), assuming you already have a way to trigger sync and store local state. The key is correctness under retries and app restarts.

Step 1: Load durable checkpoint for the scope

• Read durableSince from local storage for the scope.
• If none exists, call a snapshot bootstrap endpoint (or request deltas from since=null if supported).

Step 2: Request changes with a limit

GET /sync/changes?scope=workspace:123&since=durableSince&limit=500

• If the server returns snapshotToken and pageCursor, persist them as inProgress before applying pages.

Step 3: Apply page transactionally

• Start DB transaction.
• Apply upserts/deletes/id mappings with version checks.
• Update inProgress.pageCursor or, if using log offsets, update durableSince per page.
• Commit.

Step 4: Continue until complete

• If hasMore=true, request the next page using the cursor/snapshot token.
• Repeat apply step.

Step 5: Finalize checkpoint

• When the server indicates completion, write durableSince = nextSince in a transaction.
• Clear inProgress state.

Step 6: Retry behavior

On network failure or app restart:

• If inProgress exists with a snapshot token, resume pagination from the stored cursor.
• If no inProgress, restart from durableSince.
• Because changes are applied idempotently with version checks, re-fetching the same page should be safe.

Edge Cases and Failure Modes You Must Design For

Checkpoint too old (log retention exceeded)

If the server cannot serve deltas from the provided checkpoint, it should return a specific error code indicating the client must re-bootstrap. The client should then:

• Preserve local unsynced changes (do not wipe blindly).
• Fetch a fresh snapshot for server state.
• Reconcile local pending changes against the new baseline (often by re-uploading pending operations or re-evaluating conflicts).

Out-of-order delivery and duplicates

Mobile networks and retries can cause duplicate requests and responses. Protect yourself with:

• Per-record version checks (ignore older versions).
• Idempotent tombstone application.
• Transactional checkpoint updates.

Large backlogs

If a client has been offline for weeks, the delta could be huge. Mitigations:

• Server-side compaction for stateful entities.
• Adaptive page size: smaller pages on slow networks, larger on Wi-Fi.
• Prioritize critical entities first (for example, user profile and permissions before large lists).

Schema changes and backward compatibility

Delta payloads evolve. To avoid breaking older clients:

• Version your sync API or include a protocolVersion in requests.
• Add fields in a backward-compatible way (clients ignore unknown fields).
• When removing fields, keep them for a deprecation window or provide defaults.

Measuring Efficiency: What to Track

Delta sync should be measurable. Useful metrics include:

• Bytes transferred per sync and per entity type.
• Change density: number of changes returned vs number applied (after dedupe/version checks).
• Time to first useful data: how quickly the UI can show updated content after sync starts.
• Checkpoint lag: how far behind the client is (server revision minus client checkpoint).
• Failure rate by stage: request, apply, commit, finalize.

These metrics help you decide when to introduce compaction, patches, or different scoping strategies.