Recovering from a Stalled Replication Job #

A continuous replication that was healthy yesterday is now frozen: _scheduler/jobs reports the job as crashing or pending, changes_pending sits stubbornly above zero, and through_seq has not moved in an hour. Nothing is being copied, the backlog is growing, and the job is quietly cycling through a crash-and-back-off loop instead of failing loudly. This guide walks the triage from _scheduler/jobs and _scheduler/docs through the log, pins the root cause — an auth failure, an unreachable target, an oversized document, a crashing filter, scheduler starvation, or a full disk — and shows how to bounce the job so it returns to running and drains. It lives under Error Handling & Retry Logic. If instead the job is advancing but re-reading sequences it already copied, that is a different fault covered in Diagnosing Checkpoint Drift in CouchDB Replication.

Immediate Triage / Prerequisites #

Start by proving the job is stalled rather than idle. An idle continuous job is running with changes_pending at 0 — that is healthy and needs nothing. A stall is a job whose state is crashing or pending, or one that reads running while changes_pending stays above zero and through_seq does not move. Pull the two scheduler endpoints and read state, info, and error_count:

# per-job runtime view: state, error_count, and the last error string
curl -s http://localhost:5984/_scheduler/jobs | python3 -c "
import sys, json
for j in json.load(sys.stdin)['jobs']:
    i = j.get('info') or {}
    print(j['id'], j['database'],
          'state=', j.get('state'),
          'errors=', j.get('error_count'),
          'pending=', i.get('changes_pending'),
          'through_seq=', i.get('through_seq'))
"

# per-_replicator-document view: includes last_error and the crash reason
curl -s "http://localhost:5984/_scheduler/docs/_replicator/sensor-sync" \
  | python3 -m json.tool

_scheduler/jobs is the live in-memory view; _scheduler/docs is keyed by the _replicator document and survives across restarts, exposing info.error and last_error. A rising error_count with a repeating info.error string is the smoking gun. Cross-check the node log for the underlying exception:

grep -iE "replicator|worker_died|unauthorized|max_document_size" \
  /var/log/couchdb/couch.log | tail -n 30

Prerequisites for the steps below: Python 3.8+ with requests (pip install requests), admin credentials, and write access to the _replicator database so you can bounce the job. Restarting is cheap: because progress is checkpointed, a bounced job resumes from its last committed sequence and loses nothing.

Step-by-Step Implementation #

Work the cause down before you restart — bouncing a job whose root cause is still present just re-enters the crash loop.

1. Read the crash reason. Pull info.error/last_error from _scheduler/docs. Map the string to a class: unauthorized → 401 credential problem; econnrefused/timeout → target unreachable; max_document_size or a 413 → an oversized document; compilation_error/function_clause → a crashing filter or validate_doc_update; enospc → disk full on the target. Verify by matching the same string in couch.log.

2. Fix the root cause, not the symptom. Repair the specific fault: rotate or correct the credentials in the _replicator document; restore reachability to the target; raise max_document_size in the target’s [couchdb] config or fix the writer that produced the giant document; correct the filter or validate_doc_update function; or free disk on the target. Confirm the fix independently — for example curl -sf http://target:5984/mydb should return 200 before you proceed.

3. Locate the oversized document if that is the cause. A single document over max_document_size blocks the batch containing it forever. Find it and either shrink it or raise the limit:

   # the target rejects the write with 413; grep the doc id from the log
   grep -i "413\|max_document_size" /var/log/couchdb/couch.log | tail -n 5

   Verify the offending id is under the limit after your fix before restarting.

4. Bounce the job. For a scheduler-managed replication, force the scheduler to re-read the document by touching it, or delete and re-create it. Touching is enough because any write to the _replicator document reschedules the job:

   # re-read the doc's current _rev, then PUT it back unchanged to reschedule
   REV=$(curl -s http://localhost:5984/_replicator/sensor-sync | \
     python3 -c "import sys,json;print(json.load(sys.stdin)['_rev'])")
   curl -s -X DELETE "http://localhost:5984/_replicator/sensor-sync?rev=$REV"
   # then PUT the corrected document back

   Assert the new job appears with error_count reset and state moving toward running.

5. Confirm it drains. Poll _scheduler/jobs and watch through_seq climb and changes_pending fall toward zero. A job that returns to running and keeps error_count flat is recovered; one that re-enters crashing within seconds still has an unaddressed root cause — go back to step 1.

Complete Working Example #

The watchdog below detects a stall (state crashing/pending, or changes_pending stuck with a frozen through_seq), and restarts the job by rewriting its _replicator document — but only under capped exponential backoff so it never fuels an infinite restart loop. It is self-contained and runnable.

import sys
import time

import requests


class StallWatchdog:
    """Detect and recover a stalled CouchDB replication job.

    A stall is: state in {crashing, pending}, OR changes_pending > 0 while
    through_seq has not advanced between two polls. Recovery rewrites the
    _replicator doc, with exponential backoff to avoid a restart storm.
    """

    def __init__(self, base_url: str, repl_db: str, doc_id: str,
                 session: requests.Session, max_restarts: int = 5):
        self.base = base_url.rstrip("/")
        self.repl_db = repl_db
        self.doc_id = doc_id
        self.s = session
        self.max_restarts = max_restarts
        self._last_through = None

    def job_state(self):
        """Return (state, changes_pending, through_seq) for this doc's job."""
        r = self.s.get(f"{self.base}/_scheduler/docs/{self.repl_db}/{self.doc_id}",
                       timeout=30)
        r.raise_for_status()
        d = r.json()
        info = d.get("info") or {}
        return d.get("state"), info.get("changes_pending"), info.get("through_seq")

    def is_stalled(self, state, pending, through) -> bool:
        if state in ("crashing", "failed", "pending"):
            return True
        # running but not moving while work remains == a soft stall
        if pending and through == self._last_through:
            return True
        return False

    def restart(self):
        """Reschedule by rewriting the _replicator document unchanged."""
        url = f"{self.base}/{self.repl_db}/{self.doc_id}"
        doc = self.s.get(url, timeout=30).json()
        rev = doc.pop("_rev")
        # strip transient replicator-written fields before re-submitting
        for k in ("_replication_state", "_replication_state_time",
                  "_replication_id", "_replication_stats"):
            doc.pop(k, None)
        self.s.delete(f"{url}?rev={rev}", timeout=30).raise_for_status()
        self.s.put(url, json=doc, timeout=30).raise_for_status()

    def watch(self, poll_seconds: int = 20):
        restarts = 0
        while True:
            state, pending, through = self.job_state()
            print(f"state={state} pending={pending} through_seq={through}")
            if self.is_stalled(state, pending, through):
                if restarts >= self.max_restarts:
                    print("max restarts reached — escalating, not looping")
                    return 1
                backoff = min(2 ** restarts * 15, 300)  # cap at 5 minutes
                print(f"stalled; restart #{restarts + 1} after {backoff}s backoff")
                time.sleep(backoff)
                self.restart()
                restarts += 1
            else:
                restarts = 0  # healthy poll resets the backoff ladder
            self._last_through = through
            time.sleep(poll_seconds)


if __name__ == "__main__":
    s = requests.Session()
    s.auth = ("admin", "password")  # replace with real admin credentials
    wd = StallWatchdog("http://localhost:5984", "_replicator", "sensor-sync", s)
    sys.exit(wd.watch())

Point it at a live job and it prints each poll, and when the job stalls it waits out a growing backoff, rewrites the _replicator document to reschedule, and resets the ladder the moment the job is healthy again — recovering transient faults automatically while refusing to spin on a permanent one.

Gotchas & Edge Cases #

• An infinite crash-restart loop hides real failures. The scheduler’s own backoff means a broken job never reaches failed; it cycles crashing → pending forever with a climbing error_count. Alert on error_count rate of change, not just on the failed state, or a stall will run silently for days. Feed this into Alerting on Replication Lag with _scheduler/jobs.
• A single oversized document blocks the job forever. If one document exceeds the target’s max_document_size, every restart re-hits the same 413 on the same batch and makes no progress. Restarting will not help until you shrink the document or raise the limit — the watchdog’s max_restarts cap exists precisely so it escalates instead of looping on this.
• Restarting loses no data. Because replication checkpoints its progress, a bounced job resumes from the last committed sequence, not from zero. The only cost of a needless restart is one re-scan of the uncommitted tail — never lost writes. If a restart instead rewinds far, you are looking at checkpoint drift, not a stall.
• Idle is not stalled. A continuous job with an empty backlog sits in running with changes_pending at 0 and through_seq static — that is correct and must not trip your watchdog. Only treat a static through_seq as a stall when changes_pending is above zero.
• Scheduler starvation looks like a stall. With more replication documents than max_jobs (default 500) or a low max_churn, healthy jobs sit in pending waiting for a slot. If many jobs are pending at once with no per-job error, raise max_jobs/max_churn rather than restarting individual jobs.

Verification & Observability #

Confirm recovery at the job level and the fleet level. For the job, watch through_seq advance and changes_pending fall while error_count stays flat across several polls:

watch -n 5 "curl -s http://localhost:5984/_scheduler/jobs | python3 -c \"
import sys, json
for j in json.load(sys.stdin)['jobs']:
    i = j.get('info') or {}
    print(j['database'], j.get('state'), 'errors', j.get('error_count'),
          'pending', i.get('changes_pending'), 'seq', i.get('through_seq'))
\""

Recovery is proven when state holds at running, error_count stops climbing, and changes_pending trends to zero. For a standing signal instead of a manual watch, export error_count and the through_seq delta per job as metrics and alert on a stall the way Handling 409 Conflicts in CouchDB Replication Jobs treats a rising conflict rate — the shape of the fix is the same: watch a counter’s slope, not a single terminal state.

FAQ #

Related #

• Diagnosing Checkpoint Drift in CouchDB Replication
• Handling 409 Conflicts in CouchDB Replication Jobs
• Alerting on Replication Lag with _scheduler/jobs

Part of: Error Handling & Retry Logic