CouchDB Sync Conflict Resolution & Automation

Recovering Missing Ancestor Revisions After Compaction

A replica that was offline for a week reconnects, and replication into it stalls: _revs_diff insists the target still needs revision 4-abc…, but the source pruned that ancestor during compaction and no longer has its body. The lagging node’s ?revs_info=true marks the ancestor missing, ordinary updates land as conflicts instead of clean extensions, and the replication job sits with changes_pending refusing to drop. This page shows how to triage a missing-ancestor stall, re-seed the pruned history from a node that still holds it, and stop the pattern from recurring — the repair-side companion to Pruning Stale Revision Trees in CouchDB. It assumes you understand the branching lineage described in Revision Tree Mechanics; start there if the term “ancestor” is not yet concrete.

How a compacted ancestor stalls replication and how re-seeding fixes it Top: a source replica shows generations three, four and five along one branch; generation four is marked pruned — its body was dropped by compaction though its id is still referenced by generation five. Middle: a lagging target replica holds only generation two and asks, via _revs_diff, for generations three, four and five. The source can hand over three and five but cannot supply the pruned four, so the transfer stalls and the incoming update is filed as a conflict rather than a clean extension. Bottom: the repair path re-replicates the document from an authoritative node that still holds a complete generation-four body; afterwards _revs_diff returns an empty set and the ancestor chain from two through five is continuous. A pruned ancestor stalls the sync — until you re-seed it Source replica (after compaction) 3 4 body pruned 5 Target replica (lagging) 2 only has gen 2 _revs_diff target still needs 3 · 4 · 5 source can give 3 & 5 but not pruned 4 → stall Repair re-replicate from a node that still holds full gen-4 body then _revs_diff = { } chain 2→5 continuous Raising _revs_limit now prevents the next stall — it does not restore history already pruned. You cannot resurrect a purged ancestor locally; the only cure is to copy a surviving body from an authoritative replica. If no node still holds gen 4, that history is gone — re-seed the whole document from a full source instead.
A lagging target asks _revs_diff for generations 3, 4 and 5; the compacted source can supply 3 and 5 but not the pruned 4, so the sync stalls and the update files as a conflict. The fix is to re-replicate the document from a node that still holds a complete generation-4 body — raising _revs_limit only prevents the next stall, it never restores pruned history.

Immediate Triage / Prerequisites

Before assuming replication is broken, confirm the stall is a missing-ancestor problem and not backpressure or auth. First read the affected document’s ancestor status on the lagging target, then ask the replication protocol directly which revisions it thinks are missing:

# on the lagging target: any ancestor with status "missing" is a pruned body
curl -s "http://localhost:5984/edge_replica/sensor-42?revs_info=true" | \
  python3 -c "import sys,json;[print(x['rev'],x['status']) for x in json.load(sys.stdin)['_revs_info']]"

# ask the target what it still needs for these revs; a non-empty answer = a gap
curl -s -X POST http://localhost:5984/edge_replica/_revs_diff \
  -H "Content-Type: application/json" \
  -d '{"sensor-42":["5-c20e9f1d8a5b4e2f7c1a0b6d3e8f9a12"]}'
# {"sensor-42":{"missing":["5-c20e..."],"possible_ancestors":["2-..."]}}

A missing entry from revs_info combined with a non-empty _revs_diff response confirms the target wants revisions the source can no longer produce. Cross-check the job itself so you are not chasing a paused scheduler:

curl -s http://localhost:5984/_scheduler/docs | \
  python3 -c "import sys,json;[print(d['id'],d['state'],d.get('info',{}).get('changes_pending')) for d in json.load(sys.stdin)['docs']]"

A job stuck in running with a non-zero, non-decreasing changes_pending is the signature. Prerequisites: Python 3.8+ with requests, admin credentials on both replicas, and — this is the crux — network reach to at least one node that still holds the full history. If every replica has already compacted the ancestor away, jump to the note in the final step. For the interplay between checkpoints and stalls generally, see Error Handling & Retry Logic.

Step-by-Step Implementation

Triage first, repair second, prevent recurrence third. Each step carries a check.

  1. Identify documents with missing ancestors. Scan the candidate document set on the lagging target and flag any whose ?revs_info=true reports a missing status. That set is your repair worklist — do not touch documents whose ancestors are all available.

    curl -s "http://localhost:5984/edge_replica/sensor-42?revs_info=true" | \
  python3 -c "import sys,json;m=[x['rev'] for x in json.load(sys.stdin)['_revs_info'] if x['status']=='missing'];print('missing ancestors:',m)"

  2. Confirm the gap with _revs_diff. For each flagged document, POST its leaf revision to _revs_diff on the target. A missing array in the response is CouchDB telling you exactly which revisions replication still wants. Record the possible_ancestors — they tell you the last generation the target does hold.

  3. Locate an authoritative source that still holds the body. Query a node you trust to retain full history — a central hub, a backup replica, or one with a high _revs_limit. Use ?open_revs=all to confirm it can produce a complete leaf with an unbroken ancestor chain:

    curl -s "http://central:5984/edge_master/sensor-42?open_revs=all" \
  -H "Accept: application/json" | head -c 400

    If this node’s own revs_info shows the ancestor as available, it is a valid re-seed source. If it too reports missing, keep looking — you cannot re-seed from a node that lacks the body.

  4. Raise _revs_limit on the target going forward. Before re-seeding, widen the cap so the repaired history is not pruned again on the next compaction. This is preventive only — it does not restore anything already gone:

    curl -s -X PUT http://localhost:5984/edge_replica/_revs_limit \
  -H "Content-Type: application/json" -d '1000'

  5. Re-replicate the affected documents from the authoritative source. Run a targeted, document-filtered replication (or a fresh full replication for a badly diverged replica) from the node that holds complete history into the target, then verify with ?open_revs=all that the target now returns the leaf with no missing ancestor and _revs_diff comes back empty. If no node anywhere still holds the pruned body, you cannot recover that specific ancestor — re-seed the whole document from the most authoritative surviving copy and accept the history restarts from there.

Complete Working Example

This script scans a set of document IDs on a lagging replica, detects missing ancestors via revs_info, confirms each gap with _revs_diff, and re-seeds the affected documents by pulling complete revisions (with full _revisions history via ?revs=true) from an authoritative source and writing them back with new_edits=false so the ancestor chain is restored exactly rather than re-generated.

import sys

import requests


def missing_ancestors(db_url: str, doc_id: str, session: requests.Session) -> list:
    """Return revs whose bodies were pruned (status 'missing') on this replica."""
    resp = session.get(f"{db_url}/{doc_id}", params={"revs_info": "true"}, timeout=30)
    if resp.status_code == 404:
        return []
    resp.raise_for_status()
    return [x["rev"] for x in resp.json().get("_revs_info", []) if x["status"] == "missing"]


def confirm_gap(db_url: str, doc_id: str, leaf_rev: str, session: requests.Session) -> bool:
    """Ask the target via _revs_diff whether it still needs the leaf revision."""
    resp = session.post(f"{db_url}/_revs_diff", json={doc_id: [leaf_rev]}, timeout=30)
    resp.raise_for_status()
    return doc_id in resp.json()  # present => target is missing revisions for this doc


def fetch_full_leaf(source_url: str, doc_id: str, session: requests.Session) -> dict:
    """Pull every live leaf WITH its _revisions history from an authoritative node."""
    resp = session.get(f"{source_url}/{doc_id}",
                       params={"open_revs": "all", "revs": "true"},
                       headers={"Accept": "application/json"}, timeout=60)
    resp.raise_for_status()
    # open_revs returns a list of {"ok": {...doc with _revisions...}} envelopes
    return [row["ok"] for row in resp.json() if "ok" in row]


def reseed(target_url: str, source_url: str, doc_ids: list, session: requests.Session) -> None:
    """Restore pruned ancestors on the target by copying full history from the source."""
    to_repair = []
    for doc_id in doc_ids:
        gaps = missing_ancestors(target_url, doc_id, session)
        if gaps:
            print(f"{doc_id}: {len(gaps)} missing ancestor(s) -> repairing")
            to_repair.append(doc_id)
        else:
            print(f"{doc_id}: ancestors all available, skipping")

    for doc_id in to_repair:
        docs = fetch_full_leaf(source_url, doc_id, session)
        if not docs:
            print(f"{doc_id}: source holds no body — history unrecoverable from here")
            continue
        # new_edits=false preserves the supplied _rev/_revisions instead of minting new ones,
        # so the ancestor chain is restored exactly as it was on the source.
        resp = session.post(f"{target_url}/_bulk_docs",
                            json={"docs": docs, "new_edits": False}, timeout=60)
        resp.raise_for_status()
        still_missing = missing_ancestors(target_url, doc_id, session)
        print(f"{doc_id}: reseeded {len(docs)} leaf/leaves, "
              f"{'OK' if not still_missing else 'STILL MISSING'}")


if __name__ == "__main__":
    target = "http://admin:password@localhost:5984/edge_replica"
    source = "http://admin:password@central:5984/edge_master"
    s = requests.Session()
    reseed(target, source, ["sensor-42", "sensor-77", "sensor-91"], s)
    sys.exit(0)

Run it and each document reports whether its ancestors were intact, were repaired from the source, or are unrecoverable because no reachable node still holds the pruned body — turning a stalled sync into an explicit, auditable repair list.

Gotchas & Edge Cases

  • You cannot resurrect a purged ancestor. Pruning during compaction drops the body but keeps the id referenced by descendants; a _purge removes the id entirely. Neither is recoverable locally — the only cure is re-copying a surviving body from an authoritative replica, which is why pruning stale revision trees must be planned before it bites.
  • Raising _revs_limit does not restore pruned history. It only widens the cap for future revisions. Increase it as prevention, but never expect the old ancestors to reappear because you raised it.
  • Re-seed with new_edits=false, not a plain PUT. A normal write would mint a fresh N+1-<hash> and fork the tree, deepening the divergence. Posting the source’s documents to _bulk_docs with new_edits=false reinstates the exact _rev and _revisions chain, healing the branch instead of splitting it.
  • A too-low _revs_limit on the source is the usual root cause. If the source itself pruned the ancestor because its limit was tighter than the target’s lag, no amount of retrying on the target helps — fix the source’s limit and re-seed from a fuller node.
  • Every reachable replica missing the ancestor means it is truly gone. When open_revs on all candidate sources reports missing, stop hunting; re-seed the whole document from the most authoritative surviving copy and document that its history restarts there.

Verification & Observability

Confirm the repair at the document and the pipeline level. For the document, ?open_revs=all must return the leaf with a complete _revisions chain and ?revs_info=true must show no missing status; _revs_diff for that leaf must come back empty. For the pipeline, watch the stalled job resume draining:

# target now holds the full chain — no missing ancestors remain
curl -s "http://localhost:5984/edge_replica/sensor-42?revs_info=true" | \
  python3 -c "import sys,json;print('missing:',sum(x['status']=='missing' for x in json.load(sys.stdin)['_revs_info']))"

# _revs_diff is empty => nothing outstanding for this doc
curl -s -X POST http://localhost:5984/edge_replica/_revs_diff \
  -H "Content-Type: application/json" -d '{"sensor-42":["5-c20e9f1d8a5b4e2f7c1a0b6d3e8f9a12"]}'
# {}  (empty object = no missing revisions)

# the scheduler job is draining again, not frozen
curl -s http://localhost:5984/_scheduler/docs | \
  python3 -c "import sys,json;[print(d['id'],d['state'],d.get('info',{}).get('changes_pending')) for d in json.load(sys.stdin)['docs']]"

A healthy result is missing: 0, an empty {} from _revs_diff, and a changes_pending on the _scheduler/docs entry that finally trends toward zero. Track the count of documents reporting missing ancestors as a fleet metric; a spike right after a scheduled compaction is a direct signal that your _revs_limit is tuned too tight for how far your replicas lag.

FAQ

Can I recover a pruned ancestor without touching other nodes?

No. Once compaction drops an ancestor’s body — or _purge removes its id — that revision is gone from the local file, and nothing on that node can regenerate it. Recovery always means copying a surviving body from an authoritative replica that still holds full history. If no reachable node has it, the specific ancestor is unrecoverable and you must re-seed the whole document from the best surviving copy.

Why does an ordinary update turn into a conflict after compaction?

Because the two sides no longer share a common ancestor. When compaction on one node prunes the ancestor a lagging replica still needs, _revs_diff cannot line the histories up, so CouchDB cannot see the incoming revision as a clean extension of what the target holds. It files the write as a divergent branch — a conflict — instead. Restoring the missing ancestor from a full source lets the histories reconcile and the conflict stops appearing.

Does raising _revs_limit fix an already-stalled replication?

No. Raising the limit only changes how much history future revisions retain; it cannot bring back ancestors already pruned. To clear the current stall you must re-replicate the affected documents from a node that still holds the missing bodies, using new_edits=false so the exact revision chain is reinstated. Raise the limit as well, but treat it purely as prevention for the next cycle.

Part of: Revision Tree Mechanics