Pruning Stale Revision Trees in CouchDB #

An edge gateway with a 4 GB eMMC flash is throwing no space left on device, yet the documents it holds total a few megabytes — the bloat is revision history, thousands of superseded revision IDs and their bodies still pinned inside the .couch file. On long-lived IoT and offline-first databases the revision tree keeps growing one generation per write, and without deliberate pruning that lineage, not your data, fills the disk. This page shows how to reclaim that space safely by capping tree depth with _revs_limit, discarding superseded bodies with compaction, and — only as a last resort — hard-removing revisions with _purge, all without manufacturing new conflicts. It builds directly on Revision Tree Mechanics; if the shape of a branching _rev history is still fuzzy, read that first.

Immediate Triage / Prerequisites #

Before you touch any knob, confirm the space is actually being eaten by revision history and not by an unrelated index or attachment blob. Read the database’s storage stats and one representative document’s tree depth:

# disk_size is the on-disk .couch file; data_size is live data. A large gap = reclaimable.
curl -s http://localhost:5984/edge_telemetry | \
  python3 -c "import sys,json;d=json.load(sys.stdin);s=d['sizes'];print('disk',s['file'],'active',s['active'],'external',s['external'])"

# how deep is the history on a hot document? count the entries in _revisions.ids
curl -s "http://localhost:5984/edge_telemetry/sensor-42?revs=true" | \
  python3 -c "import sys,json;print('tree depth:',len(json.load(sys.stdin)['_revisions']['ids']))"

If file (on-disk size) dwarfs active (live data), the difference is superseded revision bodies and stale index that compaction can reclaim. If the depth on a single document is in the hundreds or thousands, _revs_limit is the lever. Prerequisites for the steps below: Python 3.8+ with requests (pip install requests), admin credentials (compaction, _purge, and setting _revs_limit all require the _admin role), and enough transient free space to hold a second copy of the file — compaction writes a new .couch before swapping it in.

Compaction needs headroom. On a nearly full edge disk it may fail mid-rewrite for lack of space; clear room or move the file elsewhere before you start, or you will trade a full disk for a corrupt compaction.

Step-by-Step Implementation #

Work from the cheapest, safest lever to the most destructive. Each step includes a check so you can confirm state before moving on.

1. Inspect the current _revs_limit. It is a per-database cap on how many trailing revision IDs each document retains (default 1000):

   curl -s http://localhost:5984/edge_telemetry/_revs_limit
   # 1000

   A value of 1000 on a database whose documents are rewritten every few seconds means CouchDB is holding up to a thousand ancestor IDs per document — far more lineage than an edge node needs.

2. Measure real tree depth before changing anything. Use ?revs_info=true to list each retained ancestor and its status; the length of that array is the current depth, and any missing entry is an already-pruned ancestor:

   curl -s "http://localhost:5984/edge_telemetry/sensor-42?revs_info=true" | \
     python3 -c "import sys,json;r=json.load(sys.stdin)['_revs_info'];print('depth',len(r),'missing',sum(x['status']=='missing' for x in r))"

   Record this number; you will compare against it after pruning to prove the depth dropped.

3. Set a sane _revs_limit. Choose a value comfortably larger than the maximum number of revisions a replica can lag behind between syncs, then PUT it as a bare integer:

   curl -s -X PUT http://localhost:5984/edge_telemetry/_revs_limit \
     -H "Content-Type: application/json" -d '100'
   # {"ok":true}

   This bounds future growth immediately — new writes trim the tree to the last 100 IDs — but it does not shrink the file on its own. Verify the read-back returns 100.

4. Run compaction to reclaim the superseded bodies. Compaction rewrites the file, dropping the bodies of non-leaf revisions beyond the limit while keeping every live leaf, including conflict branches:

   curl -s -X POST http://localhost:5984/edge_telemetry/_compact \
     -H "Content-Type: application/json"
   # {"ok":true}

   The call returns immediately; compaction runs in the background. Verify it started by finding it in _active_tasks (next step).

5. Wait for compaction to finish, then re-measure disk_size. Poll _active_tasks until no database_compaction task remains for your database, then re-read GET /db and confirm sizes.file fell toward sizes.active. Only then consider whether any individual revision must be hard-removed with _purge — and if so, plan to run the identical purge on every replica before the next replication cycle, as covered in the example below.

Complete Working Example #

This script performs the full safe sequence on one database: it snapshots disk_size, lowers _revs_limit, triggers compaction, blocks until the background task clears, and reports the reclaimed bytes. It never calls _purge automatically — purging is deliberately left as a guarded, explicit function you invoke only after coordinating across replicas.

import sys
import time

import requests


def db_sizes(db_url: str, session: requests.Session) -> dict:
    """Return the sizes block from GET /db: file (on-disk), active (live), external."""
    resp = session.get(db_url, timeout=30)
    resp.raise_for_status()
    return resp.json()["sizes"]


def set_revs_limit(db_url: str, limit: int, session: requests.Session) -> None:
    """Cap retained trailing revision ids per document. Body is a bare integer."""
    resp = session.put(f"{db_url}/_revs_limit", data=str(limit),
                       headers={"Content-Type": "application/json"}, timeout=30)
    resp.raise_for_status()
    assert session.get(f"{db_url}/_revs_limit", timeout=30).json() == limit


def compact_and_wait(db_url: str, session: requests.Session, poll: float = 2.0) -> None:
    """Trigger compaction and block until no database_compaction task remains."""
    server = db_url.rsplit("/", 1)[0]
    db_name = db_url.rsplit("/", 1)[1]
    resp = session.post(f"{db_url}/_compact",
                        headers={"Content-Type": "application/json"}, timeout=30)
    resp.raise_for_status()
    while True:
        tasks = session.get(f"{server}/_active_tasks", timeout=30).json()
        running = [t for t in tasks
                   if t.get("type") == "database_compaction" and t.get("database", "").endswith(db_name)]
        if not running:
            return
        pct = running[0].get("progress", "?")
        print(f"compacting… {pct}% complete")
        time.sleep(poll)


def purge_revision(db_url: str, doc_id: str, rev: str, session: requests.Session) -> dict:
    """DANGEROUS: hard-remove a specific revision. Advances purge_seq and breaks
    replication checkpoints. You MUST run the identical purge on every replica,
    or the revision re-replicates back as a zombie. Not called automatically."""
    resp = session.post(f"{db_url}/_purge", json={doc_id: [rev]}, timeout=60)
    resp.raise_for_status()
    return resp.json()


def reclaim(db_url: str, revs_limit: int, session: requests.Session) -> None:
    before = db_sizes(db_url, session)
    print(f"before: file={before['file']} active={before['active']}")
    set_revs_limit(db_url, revs_limit, session)   # bound future depth first
    compact_and_wait(db_url, session)             # then reclaim existing bodies
    after = db_sizes(db_url, session)
    reclaimed = before["file"] - after["file"]
    print(f"after:  file={after['file']} active={after['active']}")
    print(f"reclaimed {reclaimed} bytes ({reclaimed / max(before['file'], 1):.1%})")


if __name__ == "__main__":
    db = "http://admin:password@localhost:5984/edge_telemetry"
    s = requests.Session()
    reclaim(db, revs_limit=100, session=s)
    # Purge is intentionally manual — uncomment only after coordinating all replicas:
    # print(purge_revision(db, "sensor-42", "5-bad0...", s))
    sys.exit(0)

Run it against a live edge database and it prints the file size before and after, the percentage reclaimed, and compaction progress as it works — evidence the tree was pruned rather than the data lost.

Gotchas & Edge Cases #

• Lowering _revs_limit below a replica’s lag prunes the shared ancestor. If a peer syncs only once a day and accumulates 300 revisions in between, a limit of 100 erases the common ancestor both sides need to reconcile — CouchDB then records an ordinary update as a fresh conflict. Size the limit to the worst-case lag, not the average.
• Compaction never removes a live conflict leaf. It reclaims superseded, non-leaf bodies only. If your file stays large after compaction, you likely have many unresolved conflict branches still pinned as live leaves — resolve them (see Conflict Generation Models), don’t purge them.
• _purge resets purge_seq and desynchronizes checkpoints. Purging on one node without purging on the others means the next replication treats the purged revision as a missing update and copies it straight back — a zombie revision. Purge is an all-replicas operation or it is worse than doing nothing.
• You cannot un-purge. A purged revision’s ID and body are gone from that node; nothing but a re-copy from a replica that still holds it can bring it back. If you purged the only surviving copy, that history is unrecoverable — see Recovering Missing Ancestor Revisions After Compaction.
• Compaction is I/O heavy on constrained devices. It reads and rewrites the entire file. On a slow eMMC or SD card, schedule it during idle windows and expect elevated latency; never trigger it in a tight loop from a monitoring script.

Verification & Observability #

Confirm the reclaim worked at two levels: the file and the tree. For the file, compare sizes.file from GET /db before and after; a healthy result is file dropping close to active. For the tree, re-run ?revs_info=true on the documents you measured in triage and assert the depth is now capped at your _revs_limit:

# file shrank toward live-data size
curl -s http://localhost:5984/edge_telemetry | \
  python3 -c "import sys,json;s=json.load(sys.stdin)['sizes'];print('file',s['file'],'active',s['active'],'ratio %.2f'%(s['file']/max(s['active'],1)))"

# per-doc depth is now bounded by _revs_limit, and purge_seq is unchanged if you did not purge
curl -s "http://localhost:5984/edge_telemetry/sensor-42?revs_info=true" | \
  python3 -c "import sys,json;print('depth',len(json.load(sys.stdin)['_revs_info']))"

Track sizes.file as a gauge in your metrics pipeline; a sawtooth that climbs between compactions and drops after each is exactly what healthy pruning looks like. If purge_seq (visible in GET /db) ever advances on one node but not its peers, treat that as an alert — it is the signature of an uncoordinated purge that is about to resurrect zombie revisions across the mesh.

FAQ #

Related #

• How CouchDB Revision IDs Are Generated
• Recovering Missing Ancestor Revisions After Compaction
• Conflict Generation Models

Part of: Revision Tree Mechanics