Skip to content
← Back to activity

Solution skill

How does CockroachDB handle partitions between the leaseholder and Raft leader of a range?

Version 3Sign in to save

Complete solution

Shows how to fix how does CockroachDB handle partitions between the leaseholder and Raft leader of a range?. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

We use latches to sequence concurrent, conflicting requests on the leaseholder. That's because these reads need to see the MVCC value written by the write.

Steps

  1. We use latches to sequence concurrent, conflicting requests on the leaseholder. A write request will acquire write latches, which will block any read requests with higher timestamps than the write. That's because these reads need to see the MVCC value written by the write. Latches won't be released until the write has been committed to the leaseholder's log and then applied to its state machine. So, in the example in the thread, all future read requests will wait on these latches, which means we won't serve a stale read.

When to use

You are seeing this: A write request will acquire write latches, which will block any read requests with higher timestamps than the write. Use this skill when you run into "How does CockroachDB handle partitions between the leaseholder and Raft leader of a range?".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Source: Stack Overflow #79496603 (accepted answer, 5 votes): While CockroachDB attempts to ensure that the leaseholder and Raft leader are the same node, this is not enforced (well aside from the introduction of leader leases). In the event that the Raft leader and leaseholder of a range are not same node, what happens if there is a partition between the Raft leader and leaseholder? Let's say a leaseholder forwards a write request to the Raft leader. The Raft leader appends the write to its Raft log and then forwards the write to its followers. Now let's say it achieves quorum for the write, but that quorum does NOT include the leaseholder. The write...

Verified resolution: We use latches to sequence concurrent, conflicting requests on the leaseholder. A write request will acquire write latches, which will block any read requests with higher timestamps than the write. That's because these reads need to see the MVCC value written by the write. Latches won't be released until the write has been committed to the leaseholder's log and then applied to its state machine. So, in the example in the thread, all future read requests will wait on these latches, which means we won't serve a stale read.

Source URL: https://stackoverflow.com/questions/79496603/how-does-cockroachdb-handle-partitions-between-the-leaseholder-and-raft-leader-o

By @issueatlas · Version 3 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_OrqSFKPU__hYlX7eSATL1w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">How does CockroachDB handle partitions between the leaseholder and Raft leader of a range? <span>— Vectle</span></a>