Diagnosing Checkpoint Drift in CouchDB Replication #

Your continuous replication has started chewing through bandwidth it should not need: the job re-reads tens of thousands of changes it already copied yesterday, through_seq keeps resetting to a lower value on every restart, and documents you already resolved reappear as fresh conflicts on the target. This is checkpoint drift — the source and target no longer agree on how far replication has progressed, so CouchDB rewinds to the last sequence both sides can prove they reached and re-scans the range in between. This guide shows you how to read the _local/ checkpoint documents on each side, compare their session histories, and decide whether to accept a one-time re-scan or reset the job cleanly. It sits under Error Handling & Retry Logic; if the job is not merely re-scanning but has stopped advancing at all, see Recovering from a Stalled Replication Job instead.

Immediate Triage / Prerequisites #

Before touching the job, confirm you are looking at drift and not ordinary continuous idling. The signature is a through_seq that advances during a run but resets downward every time the job restarts, plus repeated large revisions_checked counts with a low docs_written. Pull the job state and watch for the reset:

# what sequence does the scheduler think it has committed?
curl -s http://localhost:5984/_scheduler/jobs | python3 -c "
import sys, json
for j in json.load(sys.stdin)['jobs']:
    i = j.get('info') or {}
    print(j['id'], 'through_seq=', i.get('through_seq'),
          'checkpointed=', i.get('checkpointed_source_seq'),
          'checked=', i.get('revisions_checked'),
          'written=', i.get('docs_written'))
"

# the same numbers appear here for a legacy/one-shot job
curl -s http://localhost:5984/_active_tasks | python3 -c "
import sys, json
for t in json.load(sys.stdin):
    if t.get('type') == 'replication':
        print(t.get('replication_id'), t.get('through_seq'), t.get('checkpointed_source_seq'))
"

A healthy continuous job shows checkpointed_source_seq tracking through_seq a short distance behind it and both climbing. Drift shows checkpointed_source_seq frozen or jumping backward after a restart. Prerequisites for the steps below: Python 3.8+ with requests (pip install requests), admin credentials (the _local checkpoint documents are only readable by a database admin), and the exact replication_id for the job — it is the string you just printed, and it is also the document id of the checkpoint on both databases.

The checkpoint document id and the replication id are the same value. If you change a filter, selector, query_params, or the source/target URL, the replication id changes, so CouchDB looks for a checkpoint document that does not exist and starts from sequence 0. A “sudden full re-scan” after an edit to the _replicator document is almost always this, not corruption.

Step-by-Step Implementation #

Follow these steps to read both checkpoints, diff their histories, and locate the sequence CouchDB will actually restart from.

1. Locate the replication id. Read it from _scheduler/jobs (field id) or _scheduler/docs (field replication_id). This one string names the _local checkpoint document on both endpoints:

   curl -s http://localhost:5984/_scheduler/docs/_replicator/sensor-sync \
     | python3 -c "import sys,json;print(json.load(sys.stdin)['info']['replication_id'])"
   # 6f2a0b7c9d1e4f8a2b3c4d5e6f7a8b9c

2. GET the checkpoint on the source. The checkpoint lives at _local/<replication_id>. _local documents are never replicated and never appear in the change feed, so you must query each endpoint directly:

   curl -s http://admin:pass@source:5984/mydb/_local/6f2a0b7c9d1e4f8a2b3c4d5e6f7a8b9c \
     | python3 -m json.tool

   Verify the response contains a session_id, a source_last_seq, and a history array. A 404 here means the checkpoint was deleted or never written — the next run will start from 0.

3. GET the checkpoint on the target. Repeat against the target database. The document id is identical:

   curl -s http://admin:pass@target:5984/mydb/_local/6f2a0b7c9d1e4f8a2b3c4d5e6f7a8b9c \
     | python3 -m json.tool

   Confirm both sides parsed. If either source_last_seq is a smaller value than the other, you have found the drift; the smaller value is the ceiling on where replication can safely resume.

4. Diff the history arrays by session_id. CouchDB does not simply trust the larger source_last_seq. It walks both history arrays and finds the newest session_id recorded on both sides, then resumes from that session’s recorded_seq. Compute that intersection yourself so you know the exact restart point before you act. The complete example below does this; the key is that the restart sequence is the highest recorded_seq whose session_id appears in both histories, not min(source_last_seq).

5. Decide: accept the re-scan or reset the job. If the gap is small (a few thousand sequences) and the target is healthy, do nothing — the job will re-scan once, write the missing checkpoint forward, and settle. If the gap is enormous, or one side has no common session at all (restart point 0), decide deliberately whether a full re-scan is acceptable or whether you should reset and re-seed. A re-scan is always safe (CouchDB re-checks via _revs_diff and only re-writes genuinely missing revisions) but it costs bandwidth and can resurface conflicts you already resolved.

Complete Working Example #

The tool below fetches both _local checkpoints, aligns their history arrays by session_id, reports the true restart sequence, and quantifies the drift so you can judge whether to intervene. It is self-contained and read-only — it never modifies a checkpoint.

import argparse
import sys

import requests


def get_checkpoint(base_url: str, db: str, repl_id: str, session: requests.Session):
    """Fetch the _local/<repl_id> checkpoint doc, or None if it is absent."""
    url = f"{base_url}/{db}/_local/{repl_id}"
    resp = session.get(url, timeout=30)
    if resp.status_code == 404:
        return None  # no checkpoint -> this side would restart from seq 0
    resp.raise_for_status()
    return resp.json()


def sessions_by_id(checkpoint: dict) -> dict:
    """Map session_id -> recorded_seq for every entry in a checkpoint history."""
    out = {}
    for entry in (checkpoint or {}).get("history", []):
        # recorded_seq is the source sequence committed at the end of that session
        out[entry["session_id"]] = entry.get("recorded_seq")
    return out


def restart_seq(source_cp: dict, target_cp: dict):
    """The sequence replication will actually resume from.

    It is the recorded_seq of the newest session_id present in BOTH histories.
    If nothing matches, replication restarts from the beginning (0).
    """
    if not source_cp or not target_cp:
        return 0, None
    src = sessions_by_id(source_cp)
    tgt = sessions_by_id(target_cp)
    common = [sid for sid in src if sid in tgt]
    if not common:
        return 0, None
    # history is newest-first; the first shared id is the newest agreed session
    for entry in source_cp["history"]:
        if entry["session_id"] in tgt:
            return entry.get("recorded_seq"), entry["session_id"]
    return 0, None


def main() -> int:
    ap = argparse.ArgumentParser(description="Diff CouchDB replication checkpoints")
    ap.add_argument("--source", required=True, help="e.g. http://admin:pass@src:5984")
    ap.add_argument("--target", required=True, help="e.g. http://admin:pass@tgt:5984")
    ap.add_argument("--db", required=True)
    ap.add_argument("--repl-id", required=True)
    args = ap.parse_args()

    s = requests.Session()
    source_cp = get_checkpoint(args.source, args.db, args.repl_id, s)
    target_cp = get_checkpoint(args.target, args.db, args.repl_id, s)

    src_seq = (source_cp or {}).get("source_last_seq")
    tgt_seq = (target_cp or {}).get("source_last_seq")
    print(f"source source_last_seq: {src_seq}")
    print(f"target source_last_seq: {tgt_seq}")

    resume, sid = restart_seq(source_cp, target_cp)
    print(f"agreed restart session: {sid}")
    print(f"replication will resume from sequence: {resume}")

    # quantify the drift when both sides carry numeric sequences
    if isinstance(src_seq, int) and isinstance(resume, int):
        gap = src_seq - resume
        verdict = "clean" if gap == 0 else ("minor re-scan" if gap < 5000 else "large re-scan")
        print(f"drift gap: {gap} sequences ({verdict})")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it as python checkpoint_diff.py --source http://admin:pass@src:5984 --target http://admin:pass@tgt:5984 --db mydb --repl-id 6f2a…. It prints each side’s source_last_seq, the agreed restart session and sequence, and the size of the gap replication must re-scan — the single number that tells you whether to wait it out or reset.

Gotchas & Edge Cases #

• A changed filter or selector is not drift — it is a new replication. Editing filter, selector, query_params, or either endpoint URL changes the computed replication id, so CouchDB looks for a checkpoint document under a different _local id, finds none, and restarts from 0. Expect this after any _replicator edit and schedule it for a quiet window. If you must change a filter without a full re-scan, plan it the way you would a credential change in Handling 409 Conflicts in CouchDB Replication Jobs — carefully and off-peak.
• Deleting a _local checkpoint forces a full re-scan. Some operators delete _local/<id> “to fix” a stuck job. It works, but it discards all progress and re-reads the entire change feed. Prefer bouncing the job first; delete the checkpoint only as a deliberate re-seed.
• A rebuilt target rolls its checkpoint back. Restoring a target from a snapshot, or recreating it, reverts its _local checkpoint to whatever the snapshot held while the source still records 48210. The histories then agree only at the older sequence — exactly the drift in the diagram. After any target rebuild, expect and budget for one large re-scan.
• _local documents can be compacted away. _local docs have no revision history and are not replicated, but they are stored in the database file; an aggressive external cleanup, a DELETE, or restoring an older file can lose them. Never assume the checkpoint is durable across a manual database file swap.
• Sequence values are opaque strings on a clustered node. On a CouchDB node the source_last_seq looks like 48210-g1AAAA…. Compare the whole opaque token for equality; only compare the numeric prefix for a rough “how far apart” estimate, never for correctness.

Verification & Observability #

After you let the job re-scan or reset it, confirm the checkpoints have re-converged. Re-run the diff tool and check that restart session now points at a recent session and the drift gap has collapsed to zero. Then watch the scheduler write the checkpoint forward:

# checkpointed_source_seq should now climb and stay close behind through_seq
watch -n 5 "curl -s http://localhost:5984/_scheduler/jobs | python3 -c \"
import sys, json
for j in json.load(sys.stdin)['jobs']:
    i = j.get('info') or {}
    print(j['id'], i.get('checkpointed_source_seq'), i.get('through_seq'))
\""

A converged job shows checkpointed_source_seq advancing and surviving a manual restart without dropping — the definitive proof the drift is gone. For a standing dashboard rather than a one-off check, wire the checkpointed_source_seq gap into Monitoring CouchDB Replication Checkpoints via the API and alert when the committed sequence stops moving while changes are still pending.

FAQ #

Related #

• Recovering from a Stalled Replication Job
• Handling 409 Conflicts in CouchDB Replication Jobs
• Monitoring CouchDB Replication Checkpoints via the API

Part of: Error Handling & Retry Logic