Scaling Peer-to-Peer Sync Beyond Three Nodes

Three peers in a full mesh is six directed replication jobs and it just works — so a fourth and fifth node feel free. They are not. A full peer mesh grows its connection count as N(N-1), and each of those connections is a continuous _replicator document holding an open _changes feed, a checkpoint, and a slice of your file-descriptor budget. Somewhere past the fourth node the arithmetic turns on you: jobs pile up behind the scheduler’s max_jobs ceiling, one slow peer backpressures every link that touches it, and checkpoint writes start to storm. This guide shows how to count the real link cost, tune the [replicator] scheduler, distribute jobs across nodes, and decide when to collapse the mesh into a star relay or shard it by document namespace before it collapses on its own. It builds on Sync Topology Models and picks up where a working three-node mesh from Setting Up Peer-to-Peer Sync Topologies leaves off.

The problem is that continuous replication jobs are not free background threads; they are scheduled, capped, and rotated. When every _replicator document lands on a single orchestrating node, that node’s scheduler must run all N(N-1) of them, and the default max_jobs of 500 becomes a hard wall the mesh hits around N = 23:

Total continuous replication jobs against node count. A full mesh follows N(N-1) and crosses the default max_jobs = 500 around 23 nodes when all documents sit on one orchestrator; a star relay follows 2N and barely leaves the axis.

Immediate Triage / Prerequisites

If sync has begun to lag as you added nodes, confirm you are hitting a scheduling ceiling rather than a per-link fault before you re-architect anything.

Count the jobs the scheduler is actually carrying. curl -s http://localhost:5984/_scheduler/jobs | python3 -c "import sys,json;print(json.load(sys.stdin)['total_rows'])". Compare that against max_jobs.
Read the current scheduler limits. curl -s http://localhost:5984/_node/_local/_config/replicator. Note max_jobs (default 500), max_churn (default 20), and interval (default 60000 ms). These govern how many jobs run and how fast the scheduler rotates them.
Look for jobs stuck in pending. If total_rows exceeds max_jobs, the surplus sits in pending and only runs when the scheduler rotates — that is your lag. curl -s http://localhost:5984/_scheduler/jobs | python3 -c "import sys,json;print(sum(1 for j in json.load(sys.stdin)['jobs'] if j['state']=='pending'))".
Check the file-descriptor headroom. Each continuous job holds sockets open. cat /proc/$(pgrep -f beam.smp | head -1)/limits | grep 'open files' — if the soft limit is near the job count times a few, you are near the wall.
Environment. Python 3.9+ and the standard library for the planner below; no third-party packages required.

Step-by-Step Implementation

Work through these in order — each step either buys headroom or tells you the mesh must give way to a star.

Count the required links exactly. For a full mesh, directed jobs = N(N-1). Write it down for your target N: six nodes is 30, ten is 90, twenty is 380. If that number, concentrated on the busiest node, approaches max_jobs, the mesh is already the wrong shape.
Distribute jobs across nodes. The scheduler only runs jobs whose _replicator document lives on that node. Host each node’s outbound documents on that node itself, so a full mesh of N peers puts N-1 continuous jobs on each node instead of N(N-1) on one. Verify the spread: the total_rows on every node should be roughly equal.
Tune [replicator] max_jobs deliberately. If distribution still leaves a node above the default, raise the ceiling — but size it against RAM and descriptors, not optimism. curl -X PUT http://localhost:5984/_node/_local/_config/replicator/max_jobs -d '"800"'. Each running job costs memory for its buffers and at least one socket; 800 jobs is a real resource commitment.
Raise max_churn only if rotation is starving jobs. max_churn caps how many jobs the scheduler starts or stops per interval. If you have far more jobs than max_jobs and the pending set rotates too slowly, a higher max_churn speeds turnover at the cost of more checkpoint activity. Leave interval at 60000 ms unless you have measured a reason to change it.
Decide the collapse point: star relay or namespace shard. When per-node jobs still exceed budget after distributing, stop scaling the mesh. Either collapse to a star relay — every node keeps one bidirectional link to a central node, dropping the fleet to 2N jobs, as compared in Star vs Mesh Replication Topologies: A Decision Guide — or shard by document namespace so each mesh only spans the nodes that share a selector, replacing one large mesh with several small ones.

Complete Working Example

The planner below computes the mesh job budget for N nodes, warns when the busiest node would exceed a max_jobs limit, and — when it does — emits a star-relay _replicator document set instead of the mesh. It talks to nothing; it prints a plan you review before applying.

import json
import sys
from itertools import combinations


def mesh_jobs(nodes):
    """Directed _replicator docs for a full mesh, hosted on each source node."""
    plan = []
    for a, b in combinations(nodes, 2):
        plan.append((a, _doc(a, b)))  # a->b lives on a
        plan.append((b, _doc(b, a)))  # b->a lives on b
    return plan


def star_jobs(nodes, relay):
    """Directed docs for a star relay: 2N jobs total, N-1 leaves plus relay."""
    plan = []
    for leaf in nodes:
        if leaf == relay:
            continue
        plan.append((leaf, _doc(leaf, relay)))   # push leaf -> relay
        plan.append((relay, _doc(relay, leaf)))  # pull relay -> leaf
    return plan


def _doc(src, dst, db="fleet_state"):
    return {
        "_id": f"rep__{src}__to__{dst}",
        "source": f"https://{src}:6984/{db}",
        "target": f"https://{dst}:6984/{db}",
        "continuous": True,
        "create_target": False,
    }


def busiest_node_load(plan):
    """Max number of jobs any single node must host."""
    counts = {}
    for host, _ in plan:
        counts[host] = counts.get(host, 0) + 1
    return max(counts.values()), counts


def plan_topology(nodes, max_jobs=500, relay=None):
    mesh = mesh_jobs(nodes)
    peak, _ = busiest_node_load(mesh)
    total = len(mesh)
    print(f"full mesh: {total} directed jobs, peak {peak}/node", file=sys.stderr)
    if peak <= max_jobs:
        return "mesh", mesh
    # Over budget: collapse to a star relay and report the saving.
    relay = relay or nodes[0]
    star = star_jobs(nodes, relay)
    print(
        f"WARNING: mesh peak {peak} exceeds max_jobs={max_jobs}; "
        f"collapsing to star relay '{relay}' ({len(star)} jobs total)",
        file=sys.stderr,
    )
    return "star", star


if __name__ == "__main__":
    n = int(sys.argv[1]) if len(sys.argv) > 1 else 25
    nodes = [f"node-{i:02d}.local" for i in range(n)]
    shape, plan = plan_topology(nodes, max_jobs=500)
    print(f"# chosen topology: {shape}", file=sys.stderr)
    for host, doc in plan:
        # host is the node whose _replicator db should receive this doc
        print(json.dumps({"host": host, "doc": doc}))
    sys.exit(0)

Run python3 plan.py 8 and the planner keeps the mesh: eight nodes is 56 directed jobs, seven per node, comfortably under budget. Run python3 plan.py 25 and it refuses the 600-job mesh, prints a warning, and emits a 48-job star relay instead — the same convergence for an order of magnitude less scheduler pressure.

Gotchas & Edge Cases

File descriptors run out before CPU does. Each continuous job holds sockets for its _changes feed and its writes. A node running 400 jobs can exhaust the default open-files limit and start refusing connections; raise ulimit -n and the systemd LimitNOFILE in lockstep with max_jobs.
Checkpoint storms hit when many jobs share interval. Every continuous job writes a checkpoint document periodically; hundreds of jobs writing on the same cadence produce synchronized bursts of _local writes. Distributing jobs across nodes spreads the storm; a shorter interval makes it worse, not better.
One slow node backpressures every link touching it. In a full mesh a peer that is slow to accept writes stalls the N-1 inbound jobs pointed at it, and their source nodes accumulate changes_pending. The mesh has no way to route around it — this is a structural reason to prefer a star, where a slow leaf isolates to one link.
max_jobs counts running jobs, not documents. Documents beyond the ceiling sit in pending and appear healthy in the _replicator database while never actually syncing. Always compare total_rows from _scheduler/jobs against max_jobs, not the _replicator document count.
Namespace sharding beats raising max_jobs blindly. If different node groups only share a subset of documents, a per-group selector turns one dense mesh into several sparse ones, cutting the job count without a central authority — cheaper than paying for a node that can run 800 jobs.

Verification & Observability

Confirm the plan produced a fleet that actually keeps up. The signal that scaling succeeded is a flat, sub-max_jobs job count with no growing pending set and changes_pending trending to zero:

# jobs running vs the ceiling on this node
curl -s http://localhost:5984/_scheduler/jobs | \
  python3 -c "import sys,json;d=json.load(sys.stdin);j=d['jobs'];\
print('running',sum(1 for x in j if x['state']=='running'),\
'pending',sum(1 for x in j if x['state']=='pending'))"

# backlog per job: any link that is falling behind
curl -s http://localhost:5984/_scheduler/jobs | \
  python3 -c "import sys,json;[print(j['id'],j['info'].get('changes_pending')) \
for j in json.load(sys.stdin)['jobs'] if (j['info'] or {}).get('changes_pending')]"

A healthy result has zero pending jobs and changes_pending values that shrink between polls. If pending is non-zero, you are over max_jobs and must distribute, shard, or collapse. For continuous fleet-wide visibility of these numbers, wire the scheduler metrics into Prometheus Metrics & Exporter Integration for CouchDB Replication and alert when the pending count leaves zero.

FAQ

How many continuous replication jobs can one CouchDB node sustain?

The hard cap is [replicator] max_jobs, default 500, but the practical limit is lower and set by memory and file descriptors: each running job holds socket connections and per-job buffers. A modest node comfortably runs the low hundreds; pushing toward the default 500 requires raising the open-files limit and provisioning RAM for the buffers. Measure the resident memory and descriptor use at your real job count rather than trusting the ceiling.

Should I raise max_jobs or collapse the mesh to a star?

Raise max_jobs only when distributing outbound documents across nodes still leaves a legitimate, bounded need slightly above the default. When the requirement is quadratic — a full mesh that keeps growing — raising the ceiling only delays the wall. Collapse to a star relay, which turns N(N-1) jobs into 2N, or shard by document namespace into several small meshes. Scaling the number, not the topology, is the trap.

What does max_churn control, and should I change it?

max_churn limits how many replication jobs the scheduler may start or stop in each interval, default 20 per 60000 ms. It matters only when you have more jobs than max_jobs and the scheduler is rotating the pending set through the running slots. Raising it speeds that rotation so starved jobs get their turn sooner, at the cost of more frequent job start-up and checkpoint activity. If you are not over max_jobs, leave it alone.

Part of: Sync Topology Models

Scaling Peer-to-Peer Sync Beyond Three Nodes #

Immediate Triage / Prerequisites #

Step-by-Step Implementation #

Complete Working Example #

Gotchas & Edge Cases #

Verification & Observability #

FAQ #

Related #