CouchDB Sync Conflict Resolution & Automation

Diagnosing Clock-Skew-Induced Conflict Storms

Your dashboards show a sensor reading that was 21.4 at 10:00 flipping back to 19.1 a minute later, an updated_at field marching backwards in time, and the doc_update_conflict rate on an edge fleet climbing from a handful per hour to hundreds per minute — and every one of those symptoms traces to node clocks that disagree. When last-write-wins resolution decides winners by comparing wall-clock timestamps, a single node whose NTP has drifted a few seconds into the future can win every race it enters, then lose them all when it slews back, and the churn manufactures a conflict storm. This page shows how to prove clock skew is the cause, quantify it, and stop trusting wall-clock time for ordering by moving to a Hybrid Logical Clock and a monotonic write guard. It belongs to Conflict Generation Models; if you are here because reconnecting devices spike your conflict rate rather than drifting clocks, start with Preventing Conflict Storms in Offline-First Apps instead.

How a skewed node clock inverts last-write-wins ordering A real-time axis runs left to right. Node A's clock is accurate; Node B's clock is four seconds ahead of real time. Three write events are plotted against real time: at real time 10:00:01 Node A writes reading 19.1 stamped 10:00:01, at real time 10:00:02 Node B writes reading 21.4 but stamps it 10:00:06 because its clock is ahead, and at real time 10:00:03 Node A writes reading 18.7 stamped 10:00:03. Last-write-wins compares the stamps, so B's stale 21.4 stamped 10:00:06 beats A's genuinely later 18.7 stamped 10:00:03. The document winner is therefore wrong. When chrony slews B back into sync, a subsequent comparison flips the winner again; every flip that has already replicated leaves a competing leaf that CouchDB counts as a doc_update_conflict. A 4-second skew on Node B inverts last-write-wins Node A — clock correct Node B — clock +4s ahead real time → writes 19.1 stamp 10:00:01 writes 18.7 stamp 10:00:03 latest in real time writes 21.4 stamp 10:00:06 stale, but stamped in the future LWW picks max(stamp) 10:00:06 ⟶ B wins 10:00:03 (A, discarded) winner = 21.4 (WRONG) real-latest 18.7 is dropped
When last-write-wins ranks writes by wall-clock stamp, a node running four seconds ahead wins with stale data; when its clock is corrected the winner flips back, and each replicated flip becomes a doc_update_conflict.

Immediate Triage / Prerequisites

Before you touch resolution code, prove two things: that the doc_update_conflict rate is genuinely spiking, and that node clocks disagree at the moments those conflicts appear. Start by trending the conflict counter from CouchDB’s own metrics endpoint rather than eyeballing the log:

# doc_update_conflict is exposed as a counter under the couch_db stats
curl -s http://localhost:5984/_node/_local/_stats/couchdb/document_conflicts | \
  python3 -c "import sys,json;print(json.load(sys.stdin)['value'])"

# grep the raw signature over time to see the storm build
grep -c "doc_update_conflict" /var/log/couchdb/couch.log

Now interrogate the clocks. On every node running chrony, read the current offset from true time — the field that matters is the system-clock error, not whether the daemon claims to be “synced”:

# System time offset against the reference; a healthy edge node is sub-100ms
chronyc tracking | grep -Ei "system time|last offset|leap"
# System time     : 3.812004521 seconds fast of NTP time   <-- 3.8s skew

A node reporting whole seconds fast or slow of NTP time is your poison source. Prerequisites for the steps below: Python 3.8+ with requests (pip install requests), read access to /_stats, and the ability to correlate application updated_at timestamps across nodes. One rule to fix in your head now: CouchDB’s generation counter is topological, not chronological — you cannot substitute the N in N-<hash> for a timestamp, because it counts writes on a branch, not seconds on a clock. If you are unsure how generation numbers are assigned, see How CouchDB Revision IDs Are Generated.

Step-by-Step Implementation

Follow these steps to quantify the skew, stop trusting wall-clock ordering, and add a guard so a mis-synced node can no longer poison the fleet. Each step includes a check you can run before moving on.

  1. Quantify the skew across the fleet. Collect chronyc tracking offsets from every node and record the spread. The maximum pairwise difference — not any single node’s offset — is what governs how badly last-write-wins can misorder writes:

    for host in edge-01 edge-02 edge-03; do
  printf "%s " "$host"
  ssh "$host" "chronyc tracking | awk -F': ' '/System time/{print \$2}'"
done

    If the spread exceeds your median inter-write interval, LWW ordering is unreliable by construction and the storm is expected, not anomalous.

  2. Detect non-monotonic application timestamps. Pull the conflicting leaves for a hot document and check whether updated_at ever goes backwards along the winning path. A backwards step is direct proof that a future-stamped stale write won:

    curl -s "http://localhost:5984/telemetry/sensor-42?open_revs=all" \
  -H "Accept: application/json" | \
  python3 -c "import sys,json;[print(r['ok'].get('updated_at')) for r in json.load(sys.stdin) if 'ok' in r]"

    Verify the printed timestamps are not sorted the same as the reading values you expected — disorder here confirms clock skew over a topology fault.

  3. Stop trusting wall-clock time for ordering. Replace the naive max(updated_at) comparison with a Hybrid Logical Clock. An HLC pairs a physical millisecond component with a monotonic logical counter, so two events that share (or invert) a wall-clock instant still get a total, causally-consistent order. The physical part stays close to real time for humans; the logical part guarantees monotonicity even when the physical part stalls or jumps backward.

  4. Enforce NTP discipline and prefer slew over step. Configure chrony to correct drift by slewing (gradually adjusting the tick rate) rather than stepping (jumping the clock), except at boot. A step can move updated_at backwards mid-operation; a slew never does:

    # /etc/chrony/chrony.conf — step only when >1s off AND within first 3 updates
makestep 1.0 3
# slew the rest of the time so wall-clock never jumps backwards at runtime

    Verify with chronyc tracking that steady-state corrections show as small, continuous offsets rather than discrete jumps.

  5. Add a monotonic write guard. In the write path (or a validate_doc_update function), reject any incoming write whose ordering token is not strictly greater than the one already stored. This turns a poisoned node’s stale-but-future write into a rejected 403/retry rather than a silent overwrite that replicates as a conflict. The complete example below implements both the HLC and this guard.

Complete Working Example

The script below is self-contained and runnable. It implements a Hybrid Logical Clock, samples the document_conflicts counter against measured clock offset to correlate a storm with skew, and demonstrates the monotonic guard that rejects a stale future-stamped write.

import sys
import time

import requests

MAX_DRIFT_MS = 60_000  # refuse HLC updates from a peer more than 60s ahead


class HybridLogicalClock:
    """Hybrid Logical Clock (physical ms + logical counter).

    Produces monotonically increasing, causally-consistent timestamps even
    when the wall clock stalls or jumps backward, so ordering no longer
    depends on any single node's clock being correct.
    """

    def __init__(self):
        self.phys = 0   # last physical component, in milliseconds
        self.logic = 0  # logical counter that breaks ties / covers backsteps

    def _now_ms(self) -> int:
        return int(time.time() * 1000)

    def send(self) -> str:
        """Timestamp a local event."""
        wall = self._now_ms()
        if wall > self.phys:
            self.phys, self.logic = wall, 0  # clock advanced normally
        else:
            self.logic += 1                  # clock stalled/back-stepped: bump logic
        return f"{self.phys:013d}.{self.logic:06d}"

    def recv(self, remote: str) -> str:
        """Merge a remote HLC stamp, then timestamp the receive event."""
        r_phys, r_logic = (int(x) for x in remote.split("."))
        if r_phys - self._now_ms() > MAX_DRIFT_MS:
            raise ValueError(f"peer clock {r_phys} implausibly ahead; rejecting")
        wall = self._now_ms()
        new_phys = max(self.phys, r_phys, wall)
        if new_phys == self.phys == r_phys:
            self.logic = max(self.logic, r_logic) + 1
        elif new_phys == self.phys:
            self.logic += 1
        elif new_phys == r_phys:
            self.logic = r_logic + 1
        else:
            self.logic = 0
        self.phys = new_phys
        return f"{self.phys:013d}.{self.logic:06d}"


def monotonic_guard(stored: str | None, incoming: str) -> bool:
    """Accept a write only if its HLC stamp is strictly newer than stored."""
    return stored is None or incoming > stored  # zero-padded stamps sort lexically


def sample_storm(node_url: str, session: requests.Session) -> dict:
    """Correlate the conflict counter with this node's measured clock offset."""
    stats = session.get(
        f"{node_url}/_node/_local/_stats/couchdb/document_conflicts", timeout=30
    ).json()
    # chrony exposes the offset over its own socket; here we read a cached file
    with open("/var/lib/edge/last_offset_ms") as fh:
        offset_ms = float(fh.read().strip())
    return {"conflicts": stats["value"], "clock_offset_ms": offset_ms}


if __name__ == "__main__":
    hlc = HybridLogicalClock()
    stored = hlc.send()                    # existing winning write's stamp
    time.sleep(0.01)
    fresh = hlc.send()                     # a genuinely later local write
    stale_future = f"{int(time.time() * 1000) + 4000:013d}.000000"  # +4s skewed peer

    assert monotonic_guard(stored, fresh), "a truly newer write must be accepted"
    assert not monotonic_guard(fresh, stale_future) or fresh < stale_future
    # The guard alone cannot tell a skewed future stamp from a real one — which is
    # exactly why ordering must ride the HLC, not raw wall-clock, and why NTP must
    # bound the physical component. Merge the peer through recv() instead:
    try:
        merged = hlc.recv(stale_future)
        print("merged peer stamp:", merged)
    except ValueError as exc:
        print("rejected implausible peer clock:", exc)

    print(sample_storm("http://localhost:5984", requests.Session()))
    sys.exit(0)

Run it and the HLC advances monotonically regardless of the local clock, recv rejects a peer whose physical component is implausibly far ahead, and sample_storm returns a (conflicts, clock_offset_ms) pair you can log every minute — plot the two series together and a skew-driven storm shows up as conflict spikes that track the offset curve.

Gotchas & Edge Cases

  • Generation number is topological, not chronological. It is tempting to resolve by preferring the higher N in N-<hash>, but N counts writes on a branch, not time. A device that writes twice offline reaches 3-… while a device that wrote once an hour later sits at 2-…; the higher generation is older in real time. Use an application ordering token, never the generation, for time semantics.
  • NTP step versus slew. A step correction can move the wall clock backwards at runtime, which makes updated_at non-monotonic on a single node with no replication involved at all. Constrain stepping to boot (makestep 1.0 3) and slew thereafter so timestamps only ever move forward.
  • Leap seconds smear or repeat a second. During a leap second the wall clock can repeat or skip a value; two writes one leap-second apart can tie or invert. An HLC’s logical counter absorbs this because it increments whenever the physical component fails to advance.
  • A single mis-synced node poisons the whole fleet. Because LWW is a global max, one node four seconds ahead wins every contested write across every peer it replicates to until its clock is corrected — then it loses them all, doubling the storm. Alert on per-node offset, not just fleet-average health.
  • The monotonic guard needs an atomic read-compare-write. Enforcing it in application code invites a race; prefer a validate_doc_update function so CouchDB rejects the stale write inside the same transaction that stores the document. See Filtering Replicated Writes with validate_doc_update.

Verification & Observability

Confirm the storm is over at both the fleet and document levels. For the fleet, the document_conflicts counter should flatten within one replication cycle of the clocks converging; for a document, its _conflicts array should drain as leaves are resolved:

# per-document: no live conflicting leaves should remain
curl -s "http://localhost:5984/telemetry/sensor-42?conflicts=true" | \
  python3 -c "import sys,json;print('conflicts:',json.load(sys.stdin).get('_conflicts',[]))"

# fleet: the replication jobs are draining, not stalled on retries
curl -s http://localhost:5984/_scheduler/jobs | \
  python3 -c "import sys,json;[print(j['id'],j['info'].get('changes_pending')) for j in json.load(sys.stdin)['jobs']]"

The decisive signal is correlation collapse: once every node’s chronyc tracking offset is sub-100ms and writes carry an HLC token, the conflict-rate curve should stop tracking the clock-offset curve. If conflicts persist while clocks are tight, the residue is usually genuine concurrent edits, not skew — that is a resolution-policy question for Implementing Last-Write-Wins in CouchDB and the broader Algorithm Selection for Merge guide.

FAQ

Can I just use the CouchDB generation number instead of a timestamp to order writes?

No. The generation N in an N-<hash> revision is the depth of that write on its branch — it counts accepted writes, not elapsed time. An offline device that writes three times reaches generation 4 while a device that writes once an hour later is still at generation 2, so preferring the higher generation systematically favors the busier device over the more recent one. Carry an explicit ordering token such as a Hybrid Logical Clock stamp when you need time semantics.

Why does a Hybrid Logical Clock fix this when a plain timestamp does not?

An HLC pairs a physical millisecond component with a monotonic logical counter. The physical part keeps stamps close to real time so they stay human-readable and roughly correct, while the logical part guarantees the combined stamp never goes backwards even if the wall clock stalls, steps back, or two events share an instant. That gives every write a total, causally-consistent order that survives moderate clock skew, whereas a bare timestamp inverts the moment two clocks disagree.

One node was four seconds ahead for ten minutes — why did conflicts keep coming after I fixed its clock?

While the node was skewed it won every contested write and those winners replicated out as leaves across the fleet. When you corrected the clock, subsequent comparisons flipped the winner back, creating a second wave of competing leaves for the same documents. The counter keeps climbing until every affected document’s fork is resolved and the tombstoned losers propagate. Trend the document_conflicts counter until it is flat for a full replication interval before declaring the storm over.

Part of: Conflict Generation Models