Implementing Field-Union Merge in CouchDB #

Two branches of the same CouchDB document each changed a different field — one edge worker bumped firmware_version, another rewrote network_config — and whole-document Last-Write-Wins would throw one of those edits away. You need a lossless field-level merge that keeps both. This guide implements Field-Union: for every field, keep the value from the branch that changed it most recently, so disjoint edits all survive and only genuinely concurrent edits to the same field need a tiebreak. It is the middle strategy in algorithm selection for merge, the right pick once you have confirmed via choosing between LWW, Field-Union, and CRDT merge that your conflicts are spatially separated across the document rather than colliding on one key.

Immediate Triage & Prerequisites #

Field-Union is only the correct strategy when your conflicting branches genuinely edit different fields. Confirm that before writing any merge code.

• Prove the edits are disjoint. Read a conflicted document and diff its leaves: curl 'http://localhost:5984/appdb/device-42?conflicts=true', then GET ?rev= each entry in _conflicts. If the leaves differ on separate keys, Field-Union is lossless for them. If they collide on one key, you still need a tiebreak — Field-Union does not eliminate that, it isolates it to the single contested field.
• Decide your per-field ordering source. Field-Union needs to know which branch changed a field last. The robust option is a field_updated_at sub-object stamping each field; the fallback when you lack per-field stamps is a document-level updated_at plus the highest _rev hash as a stable tiebreak. _rev ordering is topological, so it is only ever a tiebreak, never the primary signal — see revision tree mechanics.
• Confirm the retained leaves. CouchDB keeps every conflicting leaf until you tombstone it; none of this is auto-resolved. Verify with ?conflicts=true that the _conflicts array is populated.
• Environment. Python 3.10+ and requests. Run one resolver per replication partition to avoid two workers racing the same _rev.

Step-by-Step Implementation #

Each step carries a check you can run before continuing. The flow is: gather every leaf, union field by field, then commit the merged winner and tombstone the losers in one batch.

1. Fetch the winner with its conflict list. GET /appdb/{id}?conflicts=true returns the read-winner body and a _conflicts array of losing leaf revisions. Verify: assert body.get("_conflicts"), otherwise there is nothing to merge.

2. Fetch every losing leaf body. The _conflicts array holds revision IDs only. Retrieve each with GET /appdb/{id}?rev={rev} so you can read its fields and per-field stamps. Verify: the number of branch bodies equals len(_conflicts) + 1.

3. Union the fields. Walk every branch and, for each user field, keep the value whose per-field stamp is newest; break exact ties on the higher _rev hash so parallel workers agree. Track the winning stamp per field so a later branch can still displace it. Verify: assert set(merged) >= set(winner_user_fields) — the merge never drops a field that any branch held.

4. Compose the merged winner. Write the unioned fields against the current read-winner’s _rev so CouchDB extends that branch. Carry forward the merged field_updated_at map so the next conflict round has ground truth. Verify: assert merged_doc["_rev"] == winner["_rev"] before posting.

5. Commit winner plus tombstones atomically. POST /appdb/_bulk_docs with the merged winner followed by a {"_id": id, "_rev": rev, "_deleted": true} tombstone for every losing leaf. On 409 the winner _rev went stale — re-read, re-merge, and retry with backoff, as covered in handling 409 conflicts in CouchDB replication jobs. Verify: re-read with ?conflicts=true and assert _conflicts is absent.

Complete Working Example #

This self-contained resolver class performs a lossless Field-Union of every conflicting leaf. It prefers a per-field field_updated_at stamp, falls back to the document updated_at, and breaks exact ties on the higher _rev. It handles nested objects by recursive union, treats fields present on only one branch as additions to keep, and commits the merged winner plus loser tombstones in one atomic batch with bounded 409 retries.

import os
import time

import requests

RESERVED = ("_id", "_rev", "_conflicts", "_deleted", "field_updated_at")


class FieldUnionResolver:
    """Losslessly merge conflicting CouchDB leaves field by field."""

    def __init__(self, db_url: str, max_retries: int = 5):
        self.db_url = db_url.rstrip("/")
        self.max_retries = max_retries
        self.session = requests.Session()

    def _fetch_branches(self, doc_id: str) -> tuple[dict, list[dict]]:
        winner = self.session.get(
            f"{self.db_url}/{doc_id}", params={"conflicts": "true"}
        ).json()
        branches = [winner]
        for rev in winner.get("_conflicts", []):
            branches.append(
                self.session.get(f"{self.db_url}/{doc_id}", params={"rev": rev}).json()
            )
        return winner, branches

    @staticmethod
    def _field_stamp(branch: dict, key: str) -> str:
        """Per-field stamp if present, else the document-level stamp."""
        per_field = branch.get("field_updated_at", {})
        return per_field.get(key, branch.get("updated_at", "1970-01-01T00:00:00Z"))

    def _merge_pair(self, into: dict, into_ts: dict, branch: dict):
        """Union one branch's user fields into the accumulator."""
        for key, value in branch.items():
            if key in RESERVED:
                continue
            ts = self._field_stamp(branch, key)
            # Nested objects merge recursively so disjoint sub-keys both survive.
            if isinstance(value, dict) and isinstance(into.get(key), dict):
                sub_ts = {k: into_ts.get(f"{key}.{k}", "") for k in into[key]}
                self._merge_pair(into[key], sub_ts, value)
                continue
            current = into_ts.get(key)
            # Newer stamp wins; exact ties break on the higher _rev hash.
            if current is None or ts > current or (
                ts == current and branch["_rev"] > into_ts.get(f"__rev__{key}", "")
            ):
                into[key] = value
                into_ts[key] = ts
                into_ts[f"__rev__{key}"] = branch["_rev"]

    def merge(self, branches: list[dict]) -> tuple[dict, dict]:
        merged: dict = {}
        merged_ts: dict = {}
        for branch in branches:
            self._merge_pair(merged, merged_ts, branch)
        # Persist clean per-field stamps for the next conflict round.
        stamps = {k: v for k, v in merged_ts.items() if not k.startswith("__rev__")}
        return merged, stamps

    def resolve(self, doc_id: str) -> dict:
        for attempt in range(1, self.max_retries + 1):
            winner, branches = self._fetch_branches(doc_id)
            losers = winner.get("_conflicts", [])
            if not losers:
                return {"ok": True, "note": "no conflict"}

            merged, stamps = self.merge(branches)
            survivor = {
                **merged,
                "field_updated_at": stamps,
                "_id": doc_id,
                "_rev": winner["_rev"],  # extend the current read-winner branch
            }
            tombstones = [
                {"_id": doc_id, "_rev": rev, "_deleted": True} for rev in losers
            ]
            resp = self.session.post(
                f"{self.db_url}/_bulk_docs", json={"docs": [survivor, *tombstones]}
            )
            if resp.status_code != 409:
                return {"ok": True, "attempt": attempt, "fields": list(merged)}
            time.sleep(min(2 ** attempt, 30))  # stale winner _rev; re-read and retry
        raise RuntimeError(f"exhausted retries resolving {doc_id}")


if __name__ == "__main__":
    url = os.environ.get("COUCH_URL", "http://localhost:5984/appdb")
    doc = os.environ.get("DOC_ID", "device-42")
    print(FieldUnionResolver(url).resolve(doc))

Point it at a document whose leaves changed different keys and every edit survives in the merged winner; the losing leaves become tombstones and the _conflicts array clears.

Gotchas & Edge Cases #

• Nested objects need recursive union or explicit rules. A shallow field copy replaces a whole sub-object, silently dropping the other branch’s sub-keys. The resolver recurses into dicts so disjoint nested edits survive — but arrays are not safe to auto-union. Decide per field whether an array is a set (union), a log (concatenate and dedupe), or an ordered value (treat as a scalar and tiebreak).
• Concurrent edits to the same field still need a tiebreak. Field-Union is lossless only for disjoint edits. When two branches write the same key, the newer per-field stamp wins and the older value is lost for that field — exactly the Last-Write-Wins outcome, now scoped to one field. If losing either value is unacceptable, that field belongs in a CRDT, not a union.
• Deletions are ambiguous. A field absent from one branch may be a deliberate delete or simply untouched. Plain Field-Union treats absence as “no opinion” and keeps the value from the branch that has it. If you need real per-field deletes, stamp a tombstone marker (for example {"__deleted": true, "ts": ...}) rather than removing the key, so the delete carries a timestamp that can win.
• Fields added on one branch always survive. A key present on only one leaf is unioned in unconditionally — this is the lossless behaviour you want, but it means a field a user meant to remove reappears unless the removal is modelled as a stamped tombstone as above.
• Clock skew still bites the same-field tiebreak. The per-field stamp is a wall-clock value; a lagging writer can win a same-field contest it should have lost. Synchronise writers with NTP and prefer hybrid logical clocks, the same discipline implementing Last-Write-Wins in CouchDB requires.

Verification & Observability #

Confirm the merge kept everything. Per document, re-read with ?conflicts=true and assert the _conflicts key is gone, then diff the merged winner against the pre-merge leaves and assert every distinct field value is accounted for — a lossless union should equal the set of newest per-field edits. Across the pipeline, emit a fields_merged count and a same_field_tiebreak count per resolution: a rising same-field-tiebreak rate means your conflicts are no longer disjoint and Field-Union is quietly turning lossy, which is the signal to re-run choosing between LWW, Field-Union, and CRDT merge for that document class. Watch replication health with GET /_scheduler/jobs to confirm the job stays running while resolution sweeps, and schedule POST /appdb/_compact off-peak to reclaim storage from the tombstoned leaves. When a field genuinely cannot be merged safely — an array whose ordering matters, say — route the document to an auto-merge rule engine that can apply an explicit declarative rule instead of the generic union.

FAQ #

Related #

• Choosing Between LWW, Field-Union, and CRDT Merge
• Implementing Last-Write-Wins in CouchDB
• Auto-Merge Rule Engines
• Algorithm Selection for Merge

Part of: Algorithm Selection for Merge