Replication Observability & Metrics in CouchDB #

When a sync pipeline silently falls hours behind, the first question in the incident channel is always “how do we even see it?” CouchDB does not surface replication health as a single dashboard number — you assemble it from several endpoints, each exposing a different slice: live per-job progress from GET /_active_tasks, authoritative scheduler state from /_scheduler/jobs and /_scheduler/docs, node-wide counters from /_node/_local/_stats, a ready-made scrape target at /_node/_local/_prometheus, and revision-tree growth from the database info document. This section of CouchDB Replication Architecture & Revision Fundamentals is the metrics reference: what each endpoint reports, how to turn raw fields into backlog, throughput, conflict-rate and disk-bloat gauges, and how to run a collector that feeds a time-series store and alerting without hammering a resource-constrained edge node.

Configuration Schema & Required Parameters #

Observability configuration is the collector’s scrape plan plus the alert thresholds it enforces; the CouchDB side needs only that the metrics endpoints be reachable and, for the native Prometheus route, that a scrape identity exists. Deploy the replication jobs themselves through the standard _replicator document schema and monitor the checkpoints they emit as described in Monitoring CouchDB Replication Checkpoints via the API. A minimal collector configuration looks like this:

# collector.yaml — one instance targets one CouchDB node
couch_url: "http://127.0.0.1:5984"    # the node this collector observes
node: "_local"                          # resolves to the node serving the request
databases: ["iot_telemetry"]           # databases whose /db sizes to sample
scrape_interval_seconds: 15             # how often to poll the API endpoints
db_size_interval_seconds: 300           # /db sizes change slowly; sample less often
request_timeout_seconds: 10             # fail fast; a hung node must not stall the loop
max_retries: 3                          # bounded exponential backoff per endpoint
thresholds:
  backlog_changes_pending: 5000         # alert when summed pending changes exceed this
  crashed_jobs: 1                        # any crashed replication job is page-worthy
  disk_bloat_ratio: 2.0                  # sizes.file / sizes.active — compaction overdue
  conflict_rate: 0.02                    # conflicted docs / doc_count SLO ceiling

Every replication endpoint reports node-local state: _active_tasks and _stats describe only the node that answers the request, and a replication job runs on exactly one node in the target cluster, so a single collector per node is the correct topology. The table below maps each endpoint to the fields that matter and the signal they carry.

The native /_node/_local/_prometheus endpoint was added in CouchDB 3.2 and requires admin authentication on the main 5984 port; enabling [prometheus] additional_port = true also exposes an unauthenticated listener on port 17986 intended for scrapers on a trusted network. Consult the Apache CouchDB metrics documentation for the exposed series. There is no first-class “conflict count” metric — that gauge is derived, covered under Core Implementation below and in depth in Conflict-Resolution Metrics & SLOs for CouchDB Sync.

Streaming Detection / Monitoring Setup #

Before building the full collector, prove the signals are readable with a minimal poller. The snippet below reads /_scheduler/docs on a fixed interval and prints the lifecycle state and backlog of every replication job on the node — the single most useful “is anything falling behind?” view. It uses only the standard library plus requests.

import time

import requests


def poll_scheduler(couch_url: str, interval: int = 15):
    """Print state and backlog for every replication job on this node."""
    session = requests.Session()
    while True:
        resp = session.get(f"{couch_url}/_scheduler/docs", timeout=10)
        resp.raise_for_status()
        for doc in resp.json()["docs"]:
            info = doc.get("info") or {}
            # changes_pending is None until the job has computed a backlog estimate
            pending = info.get("changes_pending")
            print(f"{doc['doc_id']:<28} state={doc['state']:<10} "
                  f"pending={pending} write_failures={info.get('doc_write_failures')}")
        time.sleep(interval)


if __name__ == "__main__":
    poll_scheduler("http://localhost:5984")

Watch the state column across a few cycles: a job that flips between running and crashing with a rising error_count is retrying against a failing target, while a changes_pending that only grows means the source is producing changes faster than the job drains them. changes_pending reads as null briefly after a job starts, before the scheduler has estimated the backlog — treat None as “unknown,” not zero, so a restart never masquerades as a drained queue.

Core Implementation #

The production collector scrapes each endpoint with bounded retries, normalizes the heterogeneous field shapes, and emits a flat dictionary of gauges suitable for any time-series client. The class below computes four families of signal — backlog, throughput/failures, job health, and disk bloat — plus an optional conflict-rate sample, and logs each scrape with structured context so a failed endpoint is diagnosable from the log alone.

import logging
import time
from dataclasses import dataclass, field
from typing import Any, Optional

import requests

logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s %(levelname)s %(name)s %(message)s")
logger = logging.getLogger("metrics-collector")


@dataclass
class CollectorConfig:
    couch_url: str
    databases: list
    request_timeout: int = 10
    max_retries: int = 3
    auth: Optional[tuple] = None
    conflicts_view: Optional[str] = None  # e.g. "_design/audit/_view/conflicted"


class MetricsCollector:
    """Scrape CouchDB replication endpoints and compute observability gauges.

    One instance observes one node. All replication metrics except the
    scheduler rollup are node-local, so run a collector alongside each node.
    """

    def __init__(self, config: CollectorConfig):
        self.cfg = config
        self.session = requests.Session()
        if config.auth:
            self.session.auth = config.auth

    def _get(self, path: str) -> Any:
        """GET a path with capped exponential backoff; raise on final failure."""
        url = f"{self.cfg.couch_url}{path}"
        for attempt in range(1, self.cfg.max_retries + 1):
            try:
                resp = self.session.get(url, timeout=self.cfg.request_timeout)
                resp.raise_for_status()
                return resp.json()
            except requests.RequestException as exc:
                backoff = min(2 ** attempt, 20)
                logger.warning("scrape failed path=%s attempt=%d retry_in=%ss err=%s",
                               path, attempt, backoff, exc)
                if attempt == self.cfg.max_retries:
                    raise
                time.sleep(backoff)

    def backlog_and_throughput(self) -> dict:
        """Sum changes_pending, docs_written and write failures across jobs."""
        tasks = [t for t in self._get("/_active_tasks") if t.get("type") == "replication"]
        # changes_pending can be absent/None before the estimate is ready.
        backlog = sum(t.get("changes_pending") or 0 for t in tasks)
        return {
            "replication_backlog_changes_pending": backlog,
            "replication_docs_written_total": sum(t.get("docs_written", 0) for t in tasks),
            "replication_doc_write_failures_total":
                sum(t.get("doc_write_failures", 0) for t in tasks),
            "replication_jobs_active": len(tasks),
        }

    def job_health(self) -> dict:
        """Read node-wide replicator job gauges from _stats."""
        stats = self._get("/_node/_local/_stats")
        jobs = stats.get("couch_replicator", {}).get("jobs", {})
        # each leaf is {"value": N, "type": "gauge"}; default missing to 0.
        val = lambda k: jobs.get(k, {}).get("value", 0) or 0
        return {
            "replication_jobs_running": val("running"),
            "replication_jobs_pending": val("pending"),
            "replication_jobs_crashed": val("crashed"),
        }

    def disk_bloat(self) -> dict:
        """Compute sizes.file / sizes.active per database — compaction pressure."""
        gauges = {}
        for db in self.cfg.databases:
            info = self._get(f"/{db}")
            sizes = info.get("sizes", {})
            active = sizes.get("active") or 1  # avoid divide-by-zero on an empty db
            gauges[f"db_disk_bloat_ratio.{db}"] = round(sizes.get("file", 0) / active, 3)
            gauges[f"db_doc_count.{db}"] = info.get("doc_count", 0)
            gauges[f"db_doc_del_count.{db}"] = info.get("doc_del_count", 0)
        return gauges

    def conflict_rate(self) -> dict:
        """Derive a conflict gauge from a view that emits conflicted docs.

        CouchDB has no native conflict counter, so this samples a design view
        (map: emit(null) when doc._conflicts exists) and divides by doc_count.
        Skipped unless a conflicts_view is configured, since it is more costly.
        """
        if not self.cfg.conflicts_view:
            return {}
        gauges = {}
        for db in self.cfg.databases:
            view = self._get(f"/{db}/{self.cfg.conflicts_view}?reduce=false&limit=0")
            conflicted = view.get("total_rows", 0)
            info = self._get(f"/{db}")
            total = info.get("doc_count", 0) or 1
            gauges[f"conflict_rate.{db}"] = round(conflicted / total, 4)
            gauges[f"conflicted_docs.{db}"] = conflicted
        return gauges

    def collect(self) -> dict:
        """Run every probe and merge into one flat gauge dict."""
        gauges: dict = {}
        for probe in (self.backlog_and_throughput, self.job_health,
                      self.disk_bloat, self.conflict_rate):
            try:
                gauges.update(probe())
            except Exception:
                # One failing probe must not blank the whole scrape.
                logger.exception("probe failed: %s", probe.__name__)
        logger.info("scrape complete gauges=%d backlog=%s crashed=%s",
                    len(gauges),
                    gauges.get("replication_backlog_changes_pending"),
                    gauges.get("replication_jobs_crashed"))
        return gauges


if __name__ == "__main__":
    collector = MetricsCollector(CollectorConfig(
        couch_url="http://localhost:5984",
        databases=["iot_telemetry"],
    ))
    for name, value in sorted(collector.collect().items()):
        print(f"{name} {value}")

Two design choices matter in production. First, each probe is isolated in collect, so a single failing endpoint — a database temporarily returning 500 mid-compaction — degrades one gauge family rather than blanking the whole scrape. Second, the collector never assumes changes_pending is present; treating its absence as 0 would report a freshly restarted job as fully drained. The conflict-rate probe is deliberately gated behind an explicit view name because counting conflicted documents is heavier than the other reads, and the resolution side of that signal belongs with Revision Tree Mechanics.

The gap the backlog gauge measures is the distance between the source’s head sequence and the checkpoint the job has durably written — the diagram below shows exactly which sequence numbers bound it.

Strategy Variants & Trade-offs #

Four collection strategies dominate, and most mature deployments combine two: a lightweight poll for the gauges that need per-job labels, plus the native Prometheus endpoint for everything standard. Each trades coverage, freshness, and operational weight differently.

Polling _active_tasks is the fastest path to a backlog number but blind to a job that has crashed out of the running set entirely — for that you need /_scheduler/docs, whose state field distinguishes crashing from running and whose error_count reveals a retry loop. The native /_prometheus route is the least code and the natural backbone for standard dashboards, but it does not expose every derived gauge — notably the conflict rate and per-database bloat ratio, which is where a sidecar exporter earns its keep. That exporter is the subject of Prometheus Metrics & Exporter Integration for CouchDB Replication; the MetricsCollector above is precisely the core such an exporter wraps. Whatever the mix, keep the derived gauges consistent across strategies so an alert rule reads the same series whether it was scraped natively or computed by the collector.

Deployment & Orchestration #

Run one collector alongside each CouchDB node, not one central collector polling the whole target cluster. Replication jobs are pinned to individual nodes and _active_tasks/_stats only ever report the answering node, so a single central poller would miss every job running elsewhere and double-count nothing usefully. Drive each instance from the environment so one container image serves every node:

# Container environment (one collector replica per CouchDB node)
COUCH_URL=http://127.0.0.1:5984      # sidecar talks to its co-located node
COUCH_DATABASES=iot_telemetry
SCRAPE_INTERVAL_SECONDS=15           # 15–30s balances freshness against load
DB_SIZE_INTERVAL_SECONDS=300         # /db sizes move slowly; sample sparingly
REQUEST_TIMEOUT_SECONDS=10
CONFLICTS_VIEW=_design/audit/_view/conflicted   # optional, gated conflict rate
HEALTHCHECK_PORT=9105

Set the scrape interval between 15 and 30 seconds: below that you add measurable load to a resource-constrained edge node for little extra fidelity, since changes_pending itself is only re-estimated periodically. Sample /db sizes far less often than the job gauges — file and active sizes drift over minutes, not seconds, and reading them provokes disk I/O. Expose a /healthz endpoint that returns 200 only when the last scrape succeeded within one interval and the collector can still reach its node; a scrape loop that has silently died is worse than no monitoring because the dashboard freezes on stale-but-plausible values. Pin the collector’s own last-success timestamp so an orchestrator restarts a wedged instance rather than leaving it to report ghosts. Because the collector is stateless between scrapes, it restarts cleanly and needs no durable volume — its only state is the time-series store it writes to.

Troubleshooting & Common Errors #

FAQ #

Related #

• CouchDB Replication Architecture & Revision Fundamentals
• Prometheus Metrics & Exporter Integration for CouchDB Replication
• Monitoring CouchDB Replication Checkpoints via the API
• Conflict-Resolution Metrics & SLOs for CouchDB Sync
• Revision Tree Mechanics

Part of: CouchDB Replication Architecture & Revision Fundamentals