Continuous vs One-Shot Replication: A Trade-off Matrix #

You have a stack of CouchDB sync edges to wire up — a nightly warehouse roll-up, a live config push to a hundred field gateways, a battery-powered sensor that wakes for ninety seconds an hour — and setting every one of them to continuous: true is quietly bleeding file descriptors and scheduler slots while a few of them still lag. The continuous flag in a _replicator document is a single boolean, but it commits you to a whole operating profile: propagation latency, open-socket cost, checkpoint and restart behaviour, scheduler pressure, and recovery semantics all move together with it. This page is a decision instrument. It gives you a scenario-to-mode matrix you can read down like a lookup table, a compact pros/cons ledger, and a Python helper that picks the mode from a workload profile and posts the correct document — so you stop defaulting the whole fleet to continuous out of habit. It sits under Continuous vs One-Way Sync in CouchDB _replicator, which explains how each mode runs; this page is about which one to pick, per edge.

Immediate Triage / Prerequisites #

Before you decide anything, quantify the two numbers the decision actually turns on: how fresh the target must be, and how much steady-state connection cost you can afford. If a continuous job is already misbehaving, confirm which pressure it is hitting before you reach for a mode change. Count how many jobs the scheduler is running against its ceiling:

# how many replication jobs exist versus what the scheduler will run at once
curl -s http://localhost:5984/_scheduler/jobs | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print('total jobs:',d['total_rows'])"

# the ceiling: max_jobs is how many jobs run concurrently before others queue as 'pending'
curl -s http://localhost:5984/_node/_local/_config/replicator/max_jobs

If the job total is brushing max_jobs (default 500) and you see jobs parked in pending, every extra continuous job you add now steals a slot from one that needs to run. That is scheduler pressure, and it is the strongest single argument for demoting freshness-tolerant edges to a scheduled one-shot. Prerequisites for the code below: Python 3.8+, the requests library (pip install requests), admin-equivalent credentials for the _replicator database, and a written-down staleness budget per edge (the maximum seconds of target lag your product can absorb). Keep that budget handy — it is the first column of the matrix.

Step-by-Step Implementation #

Work each edge through these steps rather than flipping one global switch. Every step ends in a check you can run.

1. State the staleness budget. Write down the tolerated target lag in seconds for this edge. Sub-minute (dashboards, live config) points toward continuous; minutes-to-hours (nightly consolidation, metered uploads) points toward scheduled one-shot. This single number resolves most rows of the matrix on its own.

2. Price the connection cost. A continuous job holds one open _changes feed for its whole life; a one-shot job holds sockets only while it runs, then releases them. On a gateway with a tight descriptor limit, confirm the headroom before committing to an always-open feed:

   # descriptors CouchDB currently holds open on this node
   curl -s http://localhost:5984/_node/_local/_stats/couchdb/open_os_files | \
     python3 -c "import sys,json;print('open files:',json.load(sys.stdin)['value'])"

3. Check scheduler headroom. Continuous jobs occupy a scheduler slot permanently; one-shot jobs occupy one only while transferring and then free it. If total_rows from the triage step is close to max_jobs, prefer one-shot for anything that tolerates lag so live edges keep their slots.

4. Decide with the matrix. Read the scenario matrix below top-to-bottom, match the row whose freshness and link profile fit this edge, and take its recommended mode. Do not average across edges — bind a mode per edge.

5. Post the matching _replicator document. Write continuous: true for a live edge or continuous: false (the default) for a scheduled one, re-reading _rev so a redeploy updates in place. Confirm the scheduler accepted it in the mode’s healthy state — running for continuous, completed for a finished one-shot:

   curl -s http://localhost:5984/_scheduler/docs/_replicator/rep_edge_config_push | \
     python3 -c "import sys,json;print(json.load(sys.stdin)['state'])"

The decision matrix #

Read down the first two columns to find your edge, then take the mode in the third.

Pros and cons at a glance #

Complete Working Example #

The script below is self-contained and runnable. It models an edge as a WorkloadProfile, chooses the mode with the same logic as the matrix, and posts the correct _replicator document idempotently. For freshness-tolerant edges it prints a ready-to-install cron line so the one-shot job is swept on a fixed interval; for live edges it deploys a continuous job and returns.

import sys
from dataclasses import dataclass

import requests


@dataclass
class WorkloadProfile:
    """The three numbers the continuous-vs-one-shot choice turns on."""
    edge_id: str
    source: str
    target: str
    staleness_budget_s: int      # max tolerated target lag, in seconds
    link_is_metered: bool        # cellular / solar / pay-per-byte?
    scheduler_near_capacity: bool  # is job count close to max_jobs?


def choose_mode(p: WorkloadProfile) -> str:
    """Return 'continuous' or 'one_shot' using the matrix logic.

    Sub-minute freshness on an unmetered link with slots to spare -> continuous.
    Everything freshness-tolerant, metered, or slot-starved -> scheduled one-shot.
    """
    wants_live = p.staleness_budget_s < 60
    if wants_live and not p.link_is_metered and not p.scheduler_near_capacity:
        return "continuous"
    return "one_shot"


def build_doc(p: WorkloadProfile, mode: str) -> dict:
    """Assemble the _replicator document body for the chosen mode."""
    doc = {
        "_id": f"rep_{p.edge_id}",
        "source": p.source,
        "target": p.target,
        "continuous": mode == "continuous",
        "user_ctx": {"name": "sync_service", "roles": ["_admin"]},
    }
    if mode == "continuous":
        # keepalive only matters for a live feed; keep it under the path's idle timeout
        doc["heartbeat"] = 30000
    return doc


def deploy(couch_url: str, doc: dict, session: requests.Session) -> None:
    """Idempotently PUT the _replicator document, carrying _rev forward on update."""
    url = f"{couch_url}/_replicator/{doc['_id']}"
    existing = session.get(url)
    if existing.status_code == 200:
        doc = {**doc, "_rev": existing.json()["_rev"]}  # update in place, no 409
    resp = session.put(url, json=doc)
    resp.raise_for_status()


def cron_line(p: WorkloadProfile, couch_url: str) -> str:
    """Emit a crontab entry that sweeps a one-shot job on a fixed interval.

    The interval is derived from the staleness budget: never sweep less often
    than the budget allows the target to drift.
    """
    minutes = max(1, p.staleness_budget_s // 60)
    trigger = (
        f"curl -s -X POST {couch_url}/_replicator "
        f"-H 'Content-Type: application/json' "
        f"-d @/etc/couch-sync/{p.edge_id}.json"
    )
    return f"*/{minutes} * * * * {trigger}"


def provision(couch_url: str, p: WorkloadProfile, session: requests.Session) -> str:
    mode = choose_mode(p)
    deploy(couch_url, build_doc(p, mode), session)
    if mode == "one_shot":
        print(f"[{p.edge_id}] scheduled one-shot; install cron:\n  {cron_line(p, couch_url)}")
    else:
        print(f"[{p.edge_id}] continuous job deployed; watch _scheduler/docs for 'running'")
    return mode


if __name__ == "__main__":
    couch = "http://admin:pass@localhost:5984"
    s = requests.Session()
    edges = [
        WorkloadProfile("config_push", f"{couch}/device_config",
                        "https://edge.local:5984/device_config",
                        staleness_budget_s=10, link_is_metered=False,
                        scheduler_near_capacity=False),
        WorkloadProfile("night_rollup", "https://edge.local:5984/telemetry",
                        f"{couch}/telemetry_warehouse",
                        staleness_budget_s=21600, link_is_metered=True,
                        scheduler_near_capacity=False),
    ]
    chosen = {e.edge_id: provision(couch, e, s) for e in edges}
    print("modes:", chosen)
    sys.exit(0)

Run it against a live database and it deploys the live edge as a continuous job, deploys the roll-up as a one-shot, and prints the crontab line that sweeps the roll-up every six hours. For the asynchronous, many-jobs-at-once version of this provisioning loop, see Automating Continuous Sync with Python Scripts.

Gotchas & Edge Cases #

• A one-shot job still checkpoints and can resume. With use_checkpoints on (the default), a one-shot job that is interrupted mid-transfer records its progress; re-running the same document resumes from the last committed sequence instead of replaying the whole database. One-shot does not mean “start from zero every time” — it means “run to the current sequence, then exit.”
• Continuous jobs survive a node restart; scheduled one-shots do not restart themselves. Because the continuous job lives as a persistent _replicator document, the scheduler re-instantiates it after a reboot or leader election. A one-shot’s next run only happens because something external — cron, a CronJob, a timer — triggers it again. If you demote an edge to one-shot, you own its schedule.
• owner and node placement decide who runs the job. With [replicator] require_valid_user enabled the owner field records the authenticated user, and the scheduler assigns the job to a node; two provisioning agents writing the same _id race and one loses with 409. Pick a single owner per edge and derive _id deterministically so a redeploy is idempotent.
• Freshness budget in seconds, sweep interval in minutes. Cron’s finest granularity is one minute, so a sub-minute staleness budget cannot be met by a scheduled one-shot at all — that requirement forces continuous. Do not paper over a ten-second budget with a “every minute” sweep; it will miss the target by design.
• max_jobs is a hard ceiling, not a suggestion. Past max_jobs, extra jobs sit in pending and never run until a slot frees. A fleet of always-on continuous jobs can pin the ceiling; converting freshness-tolerant edges to scheduled one-shots is often the cheapest way to reclaim slots for the edges that genuinely need to be live.

Verification & Observability #

Confirm each edge is behaving as its chosen mode should. For a continuous edge, the health signal is that it stays in running with an advancing checkpoint; for a scheduled one-shot, it is that each run reaches completed inside its window and the target lag never exceeds the budget.

# continuous edge: expect state 'running' and a checkpoint that keeps moving
curl -s http://localhost:5984/_scheduler/jobs | python3 -c "
import sys, json
for j in json.load(sys.stdin)['jobs']:
    print(j['doc_id'], j['info'].get('checkpointed_source_seq'), j.get('pid') is not None)"

# scheduler headroom: pending jobs mean you are at or over max_jobs
curl -s http://localhost:5984/_scheduler/jobs | python3 -c "
import sys, json
jobs = json.load(sys.stdin)['jobs']
from collections import Counter
print(Counter(j['info'].get('error') is not None and 'error' or 'ok' for j in jobs))"

A healthy continuous edge shows a checkpointed_source_seq that climbs between polls and a live pid. A healthy one-shot fleet shows no jobs parked in pending for want of a slot. Track three metrics regardless of mode: target lag versus the staleness budget (the SLO), job count versus max_jobs (scheduler headroom), and open feeds versus the descriptor limit (resource headroom). When a continuous job flaps between crashing and running, treat it as retry tuning first — the backoff and status-code mechanics live in Error Handling & Retry Logic.

FAQ #

Related #

• Continuous vs One-Way Sync in CouchDB _replicator
• Automating Continuous Sync with Python Scripts
• The _replicator Document Schema
• Error Handling & Retry Logic

Part of: Continuous vs One-Way Sync in CouchDB _replicator