Prometheus Metrics & Exporter Integration for CouchDB Replication #

Running a fleet of CouchDB replication jobs blind is a slow-motion incident: a target falls behind, a conflict backlog swells, a worker crashes on a loop, and you learn about it from a downstream customer rather than a graph. This page shows how to make CouchDB replication observable in Prometheus two complementary ways — scraping CouchDB 3.2+'s built-in /_node/_local/_prometheus endpoint for the raw internal counters, and running a small custom exporter that turns _scheduler/jobs and _active_tasks into the higher-level gauges you actually alert on: per-job lag, changes_pending, and conflict backlog. It sits inside _replicator Configuration & Sync Pipeline Management and gives the Async Monitoring & Webhooks layer a pull-based, dashboard-friendly counterpart to its push-based events.

Configuration Schema & Required Parameters #

CouchDB 3.2 and later ships a native Prometheus endpoint. You enable it in the [prometheus] section of the server configuration (typically a file in /opt/couchdb/etc/local.d/), which opens a dedicated metrics listener; the same metrics are also reachable on the main data port under the node-local path. Alongside it you register a custom exporter as a second scrape target so Prometheus collects both the raw counters and your derived gauges. The annotated configuration below shows the CouchDB side and the corresponding prometheus.yml scrape jobs:

; /opt/couchdb/etc/local.d/prometheus.ini  — enable the native metrics endpoint
[prometheus]
additional_port = false      ; false = serve metrics only on the main 5984 port
                             ; true  = also open the dedicated port below
bind_address = 127.0.0.1     ; bind the dedicated listener to loopback in production
port = 17986                 ; default dedicated metrics port when additional_port = true

[chttpd]
bind_address = 0.0.0.0       ; the main data/metrics port stays 5984

# prometheus.yml — scrape both the native endpoint and the custom exporter
scrape_configs:
  - job_name: couchdb_native
    metrics_path: /_node/_local/_prometheus   # node-local native metrics on the main port
    scheme: http
    basic_auth:                                # the endpoint requires admin auth on 5984
      username: metrics
      password: ${COUCH_METRICS_PASSWORD}
    scrape_interval: 30s
    static_configs:
      - targets: ["couch-node-01:5984"]

  - job_name: couchdb_replication_exporter    # the custom derived-gauge exporter
    metrics_path: /metrics
    scrape_interval: 30s
    static_configs:
      - targets: ["exporter-01:9928"]

The endpoint at GET /_node/_local/_prometheus returns standard Prometheus text-format counters. _local is a literal alias that resolves to the node handling the request, so each node reports its own metrics; scrape every node in the CouchDB cluster individually rather than through a load balancer, or you will get a random node per scrape and unusable, jittery series.

The native endpoint exposes CouchDB’s internal counters — families such as couch_replicator_jobs_running, couch_replicator_jobs_crashed, couch_replicator_changes_read_failures, and the general HTTP families couchdb_httpd_requests and couchdb_httpd_status_codes. These are excellent for node-level health and request volume, but they are cluster-wide aggregates, not per-job. For per-replication-job lag and conflict backlog you need the custom exporter, because those figures live in the _scheduler/jobs JSON and in per-database conflict queries, not in the native counter set. Consult the Apache CouchDB monitoring documentation for the authoritative endpoint list before pinning label names.

Streaming Detection / Monitoring Setup #

Before wrapping anything in prometheus_client, prove you can pull the raw signals on demand. The custom exporter’s entire job is to read _scheduler/jobs on each scrape and reshape it into labelled gauges, so start with a plain read that extracts the three fields the derived gauges are built from — the job state, its changes_pending, and its through_seq:

import requests


def sample_replication_jobs(base_url: str, session: requests.Session):
    """Yield (job_id, state, changes_pending, through_seq) for every replication job.

    _scheduler/jobs is the authoritative per-job view; the most recent history
    entry carries the live state, and info carries the backlog counters.
    """
    resp = session.get(f"{base_url}/_scheduler/jobs", timeout=30)
    resp.raise_for_status()
    for job in resp.json()["jobs"]:
        info = job.get("info") or {}
        state = job["history"][0]["type"] if job.get("history") else "unknown"
        yield job["id"], state, info.get("changes_pending") or 0, info.get("through_seq")


if __name__ == "__main__":
    s = requests.Session()
    s.auth = ("admin", "password")   # _scheduler/jobs requires admin
    for job_id, state, pending, through in sample_replication_jobs("http://localhost:5984", s):
        print(f"{job_id:40s} state={state:10s} changes_pending={pending} through_seq={through}")

Run it and you get one line per job with the exact numbers the exporter will publish. This read is the contract: if changes_pending is 0 the job is caught up and its lag gauge should read 0, not go stale; if the key is absent the job has not reported yet and the gauge should be omitted rather than fabricated. Getting this reshaping right on paper first keeps the exporter itself trivial. The conflict backlog gauge comes from a second, cheaper source — a _find or a view count of documents carrying a non-empty _conflicts array on the target database — which you sample on the same scrape.

Core Implementation #

The exporter below uses prometheus_client and its custom-collector interface, which is the correct pattern for a proxy exporter: instead of setting module-level gauges from a background thread, a collector re-reads CouchDB at scrape time and yields fresh metric families. That guarantees Prometheus sees the state as of the scrape, avoids stale values when the exporter loses CouchDB, and keeps cardinality bounded to one label set per job. It adds structured logging and a short request timeout so a slow CouchDB degrades the scrape rather than hanging Prometheus.

import logging
import os
import time
from typing import Iterable

import requests
from prometheus_client import start_http_server
from prometheus_client.core import GaugeMetricFamily, REGISTRY

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger("couch-replication-exporter")


class ReplicationCollector:
    """Custom collector: re-reads CouchDB on every Prometheus scrape.

    Exposes per-job derived gauges that the native endpoint does not provide:
    changes_pending, a running/crashed state flag, and target conflict backlog.
    """

    def __init__(self, base_url: str, target_dbs: Iterable[str],
                 auth: tuple, timeout: float = 10.0):
        self.base_url = base_url.rstrip("/")
        self.target_dbs = list(target_dbs)
        self.session = requests.Session()
        self.session.auth = auth
        self.timeout = timeout

    def _scheduler_jobs(self):
        r = self.session.get(f"{self.base_url}/_scheduler/jobs", timeout=self.timeout)
        r.raise_for_status()
        return r.json()["jobs"]

    def _conflict_backlog(self, db: str) -> int:
        """Count documents with live conflicts on a target db via a _find on _conflicts."""
        query = {"selector": {"_conflicts": {"$exists": True}},
                 "fields": ["_id"], "limit": 10000, "conflicts": True}
        r = self.session.post(f"{self.base_url}/{db}/_find", json=query, timeout=self.timeout)
        r.raise_for_status()
        return len(r.json().get("docs", []))

    def collect(self):
        # ---- per-job gauges from _scheduler/jobs ----
        pending = GaugeMetricFamily(
            "couch_replication_changes_pending",
            "Documents the replication job still has to process (backlog).",
            labels=["job_id", "source", "target"])
        running = GaugeMetricFamily(
            "couch_replication_job_running",
            "1 if the replication job's latest state is running, else 0.",
            labels=["job_id"])
        try:
            for job in self._scheduler_jobs():
                info = job.get("info") or {}
                labels = [job["id"], job.get("source", ""), job.get("target", "")]
                pending.add_metric(labels, float(info.get("changes_pending") or 0))
                state = job["history"][0]["type"] if job.get("history") else "unknown"
                running.add_metric([job["id"]], 1.0 if state == "started" else 0.0)
        except requests.RequestException as exc:
            # Yield a scrape-health signal instead of crashing the endpoint.
            logger.warning("scheduler read failed: %s", exc)
            up = GaugeMetricFamily("couch_replication_exporter_up",
                                   "1 if the last CouchDB read succeeded.", value=0)
            yield up
            return
        yield pending
        yield running

        # ---- conflict backlog per target database ----
        backlog = GaugeMetricFamily(
            "couch_replication_conflict_backlog",
            "Documents with a live _conflicts array on the target database.",
            labels=["database"])
        for db in self.target_dbs:
            try:
                backlog.add_metric([db], float(self._conflict_backlog(db)))
            except requests.RequestException as exc:
                logger.warning("conflict count failed for %s: %s", db, exc)
        yield backlog

        yield GaugeMetricFamily("couch_replication_exporter_up",
                                "1 if the last CouchDB read succeeded.", value=1)


if __name__ == "__main__":
    base = os.environ.get("COUCH_URL", "http://localhost:5984")
    dbs = os.environ.get("TARGET_DBS", "aggregate_telemetry").split(",")
    auth = (os.environ.get("COUCH_USER", "admin"), os.environ.get("COUCH_PASSWORD", "password"))
    REGISTRY.register(ReplicationCollector(base, dbs, auth))
    port = int(os.environ.get("EXPORTER_PORT", "9928"))
    start_http_server(port)   # serves /metrics
    logger.info("exporter listening on :%d", port)
    while True:
        time.sleep(3600)      # the collector runs on scrape; the main thread just idles

Two design points carry the reliability. First, the collector reads on scrape, so couch_replication_changes_pending is never a stale cached number — if CouchDB is unreachable the exporter emits couch_replication_exporter_up 0 and Prometheus can alert on the exporter itself. Second, the label set is deliberately small (job_id, source, target, database): replication jobs are few and stable, so cardinality stays bounded. Never label a gauge with an opaque sequence string or a document ID — that is how a metrics backend gets flooded. The changes_pending gauge published here is exactly the signal the Alerting on Replication Lag with _scheduler/jobs rules consume, and the couch_replication_conflict_backlog gauge feeds the service-level objectives in Conflict-Resolution Metrics & SLOs for CouchDB Sync.

Strategy Variants & Trade-offs #

There is no single right way to get CouchDB replication numbers into Prometheus; the four approaches below layer rather than compete, and most production setups run the first two together.

The native endpoint is mandatory groundwork: it is free, always current, and gives you node liveness and the couch_replicator_* crash and running counts with zero code. Its limitation is that it is node-scoped and counter-oriented — it will not tell you that job X is 40,000 documents behind. The custom exporter fills exactly that gap and is where your alert rules and dashboards get their per-job signals; the cost is a service you own and must keep healthy, which is why it publishes its own _up metric. Pushgateway is the right tool only for one-shot replications that start and complete between two scrapes — a pull exporter would miss them entirely — but it is the wrong tool for continuous jobs because a crashed pusher leaves a stale value that lies until you delete it. A sidecar is the custom exporter deployed one-per-node instead of centrally; it trades more moving parts for per-node fault isolation and lets you scrape over the node’s own private interface. Choose the sidecar only when a central exporter’s blast radius (one failure blinds the whole fleet) or a strict network policy forces it.

Deployment & Orchestration #

Run the custom exporter as a small stateless container, and follow the same single-writer discipline the rest of the pipeline uses: one exporter instance per CouchDB endpoint it reads, not several pointed at the same node. Two exporters scraping the same _scheduler/jobs do not corrupt anything, but they double the read load and publish duplicate series that break sum by (job_id) aggregations unless you add an instance relabel. Drive everything through the environment so one image serves every node:

# Exporter container environment (one instance per CouchDB endpoint)
COUCH_URL=http://couch-node-01:5984
COUCH_USER=metrics
COUCH_PASSWORD=__from_secret_store__     # never bake into the image
TARGET_DBS=aggregate_telemetry,config_store
EXPORTER_PORT=9928
SCRAPE_TIMEOUT=10

Give the exporter a dedicated CouchDB account with the minimum roles needed to read _scheduler/jobs and run the _conflicts _find — do not scrape as the CouchDB admin account. Expose /metrics as its own liveness surface: Prometheus scraping it is the health check, and the couch_replication_exporter_up gauge tells you whether the exporter can still reach CouchDB even when every job is quiet. For the native endpoint, bind the dedicated 17986 listener to loopback or a private interface and let Prometheus reach it over that network only — replication metrics expose source and target hostnames and job counts that you do not want on a public interface. Set the scrape scrape_interval no tighter than the exporter’s CouchDB read can comfortably serve; a 5s interval against _scheduler/jobs on a busy coordinator adds real load for no extra signal, since changes_pending does not meaningfully change that fast. Finally, keep the exporter’s clock irrelevant to correctness: it publishes instantaneous gauges read at scrape time, so unlike a rate calculator it never depends on its own wall clock being synchronized.

Troubleshooting & Common Errors #

FAQ #

Related #

• _replicator Configuration & Sync Pipeline Management
• Async Monitoring & Webhooks
• Alerting on Replication Lag with _scheduler/jobs
• Replication Observability & Metrics in CouchDB
• Conflict-Resolution Metrics & SLOs for CouchDB Sync

Part of: _replicator Configuration & Sync Pipeline Management