Rotating Replication Credentials Without Downtime #

The security team has issued a ninety-day rotation policy, and the service account your continuous _replicator job authenticates with is about to expire — but the job has been streaming a live _changes feed for weeks and you cannot afford a gap while it re-authenticates. Rotate carelessly and you get the worst outcome: the moment the old secret is revoked, the running worker’s next request returns 401, the job drops into crashing, and sync stalls until someone notices. This page shows the zero-downtime sequence: keep the new credential valid alongside the old one, patch the _replicator document so its job restarts on the new secret and resumes from checkpoint, verify it is back to running, and only then revoke the old credential. It also fixes the root-cause mistake — credentials embedded in the source/target URL, where they leak into logs — by moving them into the request auth object and headers. It builds directly on The _replicator Document Schema, whose document is the thing you are about to patch under load.

Immediate Triage / Prerequisites #

Before rotating anything, find out how the current job authenticates and whether the secret is leaking. The single most common finding is a credential baked straight into the source or target URL, which CouchDB logs verbatim on every connection error. Inspect the live document and grep the log for the exposure:

# read the deployed job body; look for user:pass@ inside source/target URLs
curl -s http://localhost:5984/_replicator/rep_edge_to_central | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print('source:',d.get('source'));print('target:',d.get('target'))"

# credentials embedded in a URL surface in the log on any connection failure
grep -Ei "https?://[^ ]*:[^ @]*@" /var/log/couchdb/couch.log | tail -n 5

If the URL carries user:pass@, that is the first thing to fix: move authentication into the source/target object’s auth field (for basic auth) or its headers (for a bearer token), where it is not echoed into log lines. Confirm the job is presently healthy before you touch it — you want a clean running baseline so any post-rotation change of state is unambiguous:

curl -s http://localhost:5984/_scheduler/docs/_replicator/rep_edge_to_central | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print(d['state'], d.get('info',{}).get('error'))"

Prerequisites for the steps below: Python 3.8+, the requests library (pip install requests), admin-equivalent access to the _replicator database, and the ability to mint and revoke the credential the job uses (a CouchDB user document, an upstream auth provider’s token, or a reverse-proxy secret). One rule frames everything: updating a _replicator document restarts its job. That restart is the mechanism you exploit to swap credentials, and it is also the reason ordering matters.

Step-by-Step Implementation #

Follow the sequence in order. Each step has a check; do not proceed to revocation until the verification step passes.

1. Move credentials out of the URL. Rewrite source/target as objects that carry the secret in auth or headers, not in the URL. Basic auth goes in auth.basic; a bearer token goes in an Authorization header:

   {
     "target": {
       "url": "https://central.example:5984/telemetry",
       "auth": {"basic": {"username": "sync_svc", "password": "OLD_SECRET"}}
     }
   }

   Verify the rewritten document still reaches running before you treat this as your baseline — this move itself restarts the job.

2. Create the new credential, valid alongside the old. Provision the replacement secret and confirm both authenticate before changing the job. The old one must keep working through the patch; the new one must already work at the moment of the patch:

   # both should return 200 against a protected endpoint
   curl -s -o /dev/null -w "old:%{http_code}\n" -u sync_svc:OLD_SECRET https://central.example:5984/telemetry
   curl -s -o /dev/null -w "new:%{http_code}\n" -u sync_svc:NEW_SECRET https://central.example:5984/telemetry

3. Patch the _replicator document with the new secret. Re-read the current _rev, swap only the credential field, and PUT it back. The scheduler notices the document changed, tears down the old worker, and starts a fresh one — which resumes from the last checkpoint rather than replaying, because use_checkpoints is on. Do not delete and recreate the document; an in-place update preserves the job’s identity and checkpoint.

4. Verify the job returned to running. Poll _scheduler/docs until the state settles back on running with no error. This is the gate: the restart must succeed on the new credential before you revoke anything.

   curl -s http://localhost:5984/_scheduler/docs/_replicator/rep_edge_to_central | \
     python3 -c "import sys,json;d=json.load(sys.stdin);print(d['state'], d.get('info',{}).get('error'))"
   # expect: running None

5. Revoke the old credential. Only now — with the job confirmed running on the new secret and its checkpoint advancing — retire the old credential at the auth provider. Because you stayed inside the overlap window, no request was ever made with a dead secret.

Complete Working Example #

The script below performs the whole rotation as one operation: it reads the job, patches the credential in place, and blocks until the scheduler reports running again (raising if the job crashes on the new secret, so a bad credential aborts the rotation instead of silently stalling sync). Revoking the old secret is deliberately left to the caller after this returns — that ordering is the entire point.

import sys
import time

import requests


def load_doc(base_url: str, doc_id: str, session: requests.Session) -> dict:
    """Fetch the current _replicator document, including its _rev."""
    resp = session.get(f"{base_url}/_replicator/{doc_id}")
    resp.raise_for_status()
    return resp.json()


def swap_credential(doc: dict, endpoint: str, username: str, new_secret: str) -> dict:
    """Return a copy of the doc with the credential replaced on source or target.

    Keeps the secret in the auth object (or an Authorization header) rather than
    in the URL, so it never lands in a log line on a connection error.
    """
    patched = dict(doc)
    for side in ("source", "target"):
        val = patched.get(side)
        if isinstance(val, dict) and val.get("url", "").startswith(endpoint):
            new_side = dict(val)
            new_side["auth"] = {"basic": {"username": username, "password": new_secret}}
            new_side.pop("headers", None)  # drop any stale Authorization header
            patched[side] = new_side
    return patched


def wait_running(base_url: str, doc_id: str, session: requests.Session,
                 timeout: int = 120, interval: int = 3) -> str:
    """Block until the restarted job is back to 'running'.

    Raises on 'crashing'/'failed' (usually a 401 from a bad new secret) so the
    caller does NOT revoke the old credential on a broken rotation.
    """
    url = f"{base_url}/_scheduler/docs/_replicator/{doc_id}"
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        info = session.get(url).json()
        state = info.get("state")
        if state == "running":
            return state
        if state in ("crashing", "failed"):
            err = info.get("info", {}).get("error")
            raise RuntimeError(f"rotation broke the job ({state}): {err!r}")
        time.sleep(interval)
    raise TimeoutError(f"{doc_id} did not return to running within {timeout}s")


def rotate(base_url: str, doc_id: str, endpoint: str, username: str,
           new_secret: str, session: requests.Session) -> str:
    """Patch the job onto the new credential and wait for it to resume.

    The old credential must still be valid for the whole of this call; revoke
    it only after this returns 'running'.
    """
    doc = load_doc(base_url, doc_id, session)
    patched = swap_credential(doc, endpoint, username, new_secret)
    # PUT with the current _rev so the scheduler updates the job in place and
    # restarts it, resuming from the existing checkpoint (use_checkpoints on).
    resp = session.put(f"{base_url}/_replicator/{doc_id}", json=patched)
    resp.raise_for_status()
    return wait_running(base_url, doc_id, session)


if __name__ == "__main__":
    couch = "http://admin:pass@localhost:5984"
    s = requests.Session()
    state = rotate(
        base_url=couch,
        doc_id="rep_edge_to_central",
        endpoint="https://central.example:5984/telemetry",
        username="sync_svc",
        new_secret="NEW_SECRET",
        session=s,
    )
    print(f"job back to {state!r}; safe to revoke the OLD credential now")
    sys.exit(0)

Run it with the old credential still live and it patches the job, waits out the restart, and prints the go-ahead to revoke — only once the scheduler confirms running. For the deployment side of this document (deterministic _id, idempotent PUT, owner handling), see The _replicator Document Schema; for how these credentials fit the broader trust model, see Security Boundaries in CouchDB Replication.

Gotchas & Edge Cases #

• Revoking the old secret before the restart confirms kills sync. If you retire the old credential while the worker is still mid-restart or has not yet picked up the new document, its next feed request authenticates with a dead secret, returns 401, and the job drops to crashing. Always gate revocation on a confirmed running state, as the script does — never on a timer.
• A bad new credential lands the job in crashing with a 401. If the replacement secret was mistyped or lacks the target’s roles, the restarted worker cannot authenticate and the scheduler parks it in crashing, retrying with backoff. wait_running raises in that case so you do not revoke the old credential — leaving the old secret live is what lets you roll back by patching the document back.
• The replicator caches the document; the change is not instantaneous. The scheduler reacts to the _replicator change and cycles the worker, but there is a short window before the new worker is fully live. That is exactly why the overlap window exists — do not assume the swap is atomic the instant your PUT returns 200.
• Secrets in the document versus a secrets store. The credential you PUT lives in the _replicator database and is readable by anyone with access to it, and it persists in that document’s revision history. Prefer a short-lived bearer token in headers, or reference an external secret via a reverse proxy, over a long-lived password sitting in the document. Filtering who may write such documents at all is covered by Filtering Replicated Writes with validate_doc_update.
• Never put the secret back in the URL to “save a field.” A credential in source/target as https://user:pass@host is logged on every connection error and stored in plaintext in the document. Keep it in auth/headers even though it is one more line of JSON.

Verification & Observability #

Confirm the rotation both took hold and left no stall behind. At the job level, the checkpoint must keep advancing across the restart; at the fleet level, no job should be sitting in crashing with an auth error after the revoke.

# checkpoint should be advancing again after the restart, not frozen
curl -s http://localhost:5984/_scheduler/jobs | python3 -c "
import sys, json
for j in json.load(sys.stdin)['jobs']:
    if j.get('doc_id') == 'rep_edge_to_central':
        print('seq:', j['info'].get('checkpointed_source_seq'), 'errors:', j.get('error_count', 0))"

# after revoking the old secret, no job should be crashing on a 401
curl -s http://localhost:5984/_scheduler/docs | python3 -c "
import sys, json
for d in json.load(sys.stdin)['docs']:
    err = (d.get('info') or {}).get('error')
    if d['state'] == 'crashing' and err:
        print('CRASHING:', d['doc_id'], err)"

A healthy result is a checkpointed_source_seq that climbs after the restart and an empty crashing list once the old credential is gone. Emit two metrics around every rotation: the job’s state-transition count (a spike beyond the single expected restart means the new credential is flapping) and the auth-error rate at the target (401s should be zero after the overlap closes). If a 401 appears after you revoked, the worker was still on the old secret when you pulled it — patch the document again to force another restart onto the new credential. When rotating credentials on many intermittently connected edges at once, sequence the rotations so each edge’s overlap window is respected, following the low-power patterns in Configuring _replicator for IoT Edge Nodes.

FAQ #

Related #

• The _replicator Document Schema
• Security Boundaries in CouchDB Replication
• Filtering Replicated Writes with validate_doc_update
• Configuring _replicator for IoT Edge Nodes

Part of: The _replicator Document Schema