Replicating Through a Firewall with Pull-Only Sync #

Your central CouchDB has a public address and your edge nodes do not — they sit behind NAT, a cellular gateway, or a corporate firewall that permits outbound connections but drops every inbound one. So the obvious wiring fails on contact: a job on the central side that names the edge as its source or target can never open the socket, and it parks in crashing forever. The fix is to invert who dials whom. In CouchDB the node that holds the _replicator document is the one that opens the outbound HTTPS connection, so every job that touches an unreachable edge must live on that edge and reach out to the reachable side. This guide shows how to choose the direction, host the job on the correct node, secure the single permitted connection with TLS and auth, and run bidirectional sync when the firewall only ever lets the edge dial out. It sits under Sync Topology Models, which frames each replication edge as a directed, configurable artifact.

The one principle that resolves every direction question: replication is driven by the node hosting the _replicator document, and that node makes the outbound connection to the remote endpoint. The firewall only ever sees a connection opened from the edge, yet data still crosses in both directions over those edge-initiated connections:

Immediate Triage / Prerequisites #

If a job that touches an edge is stuck, first prove that the failure is the connection direction and not credentials, then confirm which side can dial which.

• Read why the job is failing. curl -s http://central:5984/_scheduler/jobs | python3 -c "import sys,json;[print(j['id'],j['state'],(j.get('info') or {}).get('error')) for j in json.load(sys.stdin)['jobs']]". A connection-refused or timeout on an edge endpoint, from a job hosted on the central side, is the signature of a wrong-direction document.
• Confirm the connectivity direction empirically. From the edge, curl -skm 5 https://central.example.com:6984/ should return the welcome document. From the central side, the same curl to the edge will hang or refuse — proof the edge must be the initiator.
• Confirm outbound egress is actually allowed. Some firewalls block arbitrary outbound ports too. Verify the edge can reach the central CouchDB port (6984 for TLS) specifically, not just port 443.
• Verify credentials on the reachable side. The edge authenticates against the central node, so the replication user must exist there: curl -sk https://central.example.com:6984/_users/org.couchdb.user:edge_sync. Scope it to a replication role, never a server admin. The full field reference is The _replicator Document Schema.
• Environment. Python 3.9+ and requests (pip install requests) to POST the documents, plus a CA the edge trusts for the central node’s certificate.

Step-by-Step Implementation #

Each step ends with a check so you can confirm direction and reachability before the job goes continuous.

1. Identify the reachable side. The side with a stable, dialable address is the endpoint; the side that can only open outbound connections is the initiator. Behind NAT the edge is always the initiator. Assert it: the edge’s curl to the central node succeeds and the reverse hangs.

2. Host every edge-touching document on the edge. Create the _replicator documents in the edge node’s own _replicator database, never the central one. This is the decision that makes the outbound connection originate from the side the firewall permits — the same host-locality rule that governs job placement in Setting Up Peer-to-Peer Sync Topologies.

3. Write the pull document to receive. For the edge to receive central’s changes, its document names the remote as source and the local database as target. Set continuous: true so the outbound connection stays live. Verify it reaches running: curl -s http://localhost:5984/_scheduler/jobs on the edge shows the job draining.

4. Add a push document to send. For the edge to send its own writes up, a second document on the edge names the local database as source and the remote as target. Both documents dial out from the edge, so the firewall stays satisfied while data now moves both ways. Confirm the central node receives new revisions with ?conflicts=true sanity checks.

5. Secure the single open direction. Because exactly one connection direction exists, harden it: use https:// endpoints, pin or properly validate the central node’s certificate, and put credentials in the document’s auth object rather than the URL. Confirm TLS with curl -sv https://central.example.com:6984/ 2>&1 | grep -i 'SSL connection' from the edge.

Complete Working Example #

The script below runs on the edge and provisions both directions as edge-initiated jobs. It writes a pull document and a push document into the local _replicator database, each pointing at the reachable central node with TLS and role-scoped auth. It never asks the central node to dial the edge.

import sys

import requests

EDGE_LOCAL = "http://localhost:5984"          # this edge's own CouchDB
CENTRAL = "https://central.example.com:6984"   # the reachable, dialable side
DB = "fleet_state"
# Credentials for the replication user that exists ON THE CENTRAL node.
CENTRAL_AUTH = {"username": "edge_sync", "password": "s3cret-rotate-me"}


def pull_doc() -> dict:
    """Edge receives central's changes: remote is source, local is target."""
    return {
        "_id": f"pull__{DB}__from_central",
        "source": {"url": f"{CENTRAL}/{DB}", "auth": {"basic": CENTRAL_AUTH}},
        "target": {"url": f"{EDGE_LOCAL}/{DB}"},  # local: no outbound auth needed
        "continuous": True,       # hold the outbound connection open
        "create_target": False,   # the db must already exist on the edge
    }


def push_doc() -> dict:
    """Edge sends its writes up: local is source, remote is target."""
    return {
        "_id": f"push__{DB}__to_central",
        "source": {"url": f"{EDGE_LOCAL}/{DB}"},
        "target": {"url": f"{CENTRAL}/{DB}", "auth": {"basic": CENTRAL_AUTH}},
        "continuous": True,
        "create_target": False,
    }


def install(doc: dict, session: requests.Session) -> None:
    """PUT the _replicator document into the edge's local _replicator db."""
    url = f"{EDGE_LOCAL}/_replicator/{doc['_id']}"
    # Fetch any existing rev so re-running is idempotent, not a 409.
    head = session.head(url, timeout=15)
    if head.status_code == 200:
        doc["_rev"] = head.headers["ETag"].strip('"')
    resp = session.put(url, json=doc, timeout=15)
    resp.raise_for_status()
    print(f"installed {doc['_id']} -> {resp.json()['rev']}")


def verify(session: requests.Session) -> None:
    """Both jobs should reach 'running' from the edge side only."""
    jobs = session.get(f"{EDGE_LOCAL}/_scheduler/jobs", timeout=15).json()
    for j in jobs["jobs"]:
        info = j.get("info") or {}
        print(j["id"], j["state"], info.get("error"))


if __name__ == "__main__":
    s = requests.Session()
    install(pull_doc(), s)   # receive: source=central
    install(push_doc(), s)   # send: target=central
    verify(s)
    sys.exit(0)

Run it on the edge and both jobs appear in the edge’s _scheduler/jobs as running, each holding an outbound connection to the central node. Nothing on the central side ever needs to know the edge’s address — a strict security win covered in Security Boundaries in CouchDB Replication.

Gotchas & Edge Cases #

• Continuous pull needs a persistently reachable central endpoint. A continuous job holds a long-lived connection to the _changes feed; if the firewall or a stateful NAT drops idle outbound connections, the job cycles through reconnects. Confirm the NAT keep-alive window is longer than CouchDB’s heartbeat, or tune the connection to send heartbeats often enough to stay open.
• A forward proxy changes the endpoint, not the direction. If the edge egresses through an HTTP proxy, point the job at the proxy per your deployment’s proxy configuration; the edge is still the initiator. A transparent proxy that terminates TLS will break certificate validation unless you trust its CA.
• Certificate pinning bites on rotation. Pinning the central node’s certificate on every edge is a strong defense, but it turns a routine certificate renewal into a fleet-wide outage if you forget to re-pin. Pin the issuing CA rather than the leaf certificate, or automate the re-pin, so renewals do not strand edges.
• Do not put credentials in the URL. A source like https://user:pass@central... leaks the password into logs and the _replicator document body. Use the auth object as the example does, and rotate it on a schedule.
• create_target: false protects the constrained side. With a typo in a database name, true silently mints an empty database on a memory-limited edge; false fails the job loudly so you notice.

Verification & Observability #

Confirm both directions converge and that the only connections are edge-initiated. From the edge, both jobs should be running with a shrinking backlog; from the central side, you should see incoming replication activity but no job of its own:

# on the edge: both jobs running, draining, no error
curl -s http://localhost:5984/_scheduler/jobs | \
  python3 -c "import sys,json;[print(j['id'],j['state'],(j.get('info') or {}).get('changes_pending')) \
for j in json.load(sys.stdin)['jobs']]"

# on the central node: it hosts no edge-facing jobs of its own
curl -s http://central:5984/_scheduler/jobs | \
  python3 -c "import sys,json;print('central jobs:',json.load(sys.stdin)['total_rows'])"

A healthy result is two running jobs on the edge with changes_pending trending to zero and, ideally, zero edge-facing jobs on the central node — proof the wiring respects the firewall. If the edge job flaps between running and crashing with connection-reset errors, the NAT is timing out idle connections; shorten the heartbeat or lengthen the NAT window. Watch the central node’s active-task log for the incoming replication IDs to confirm the push half is landing.

FAQ #

Related #

• Setting Up Peer-to-Peer Sync Topologies
• Security Boundaries in CouchDB Replication
• The _replicator Document Schema

Part of: Sync Topology Models