CodeWithYash
DSA
Spring Boot
System Design
Series
Articles
Leaderboard
Sign In

Subscribe to Newsletter

Get the latest articles, DSA sheets, and system design challenges delivered to your inbox.

<CODEWITHYASH/>

The ultimate sanctuary for software engineers seeking deep mastery of core technical principles.

Library

  • Articles
  • DSA Sheets
  • Roadmaps

Connect

  • GitHub
  • LinkedIn
  • About
  • Contact

Legal

  • Privacy
  • Terms
  • Support

© 2026 CodeWithYash Global Systems.

Status: OperationalVersion: 4.2.0
System Design
OverviewTheory lessonsHLDLLD
All SeriesSeries
Distributed Systems

System Design Series

0%

0 of 33 topics completed

Progress
Idempotency & Safe RetriesTheory
25 min
Time, Clocks & Ordering in Distributed SystemsTheory
25 min
Distributed Consensus: Raft & PaxosTheory
30 min
Distributed Locking & Monotonic FencingTheory
25 min
Distributed Transactions: 2PC, Saga & Transactional OutboxTheory
30 min
Disaster Recovery & Multi-Region TopologiesTheory
25 min
System DesignDistributed Systems Mechanics & ReliabilityDistributed Consensus: Raft & Paxos

Complex Systems · Phase 3: Distributed Mechanics

Distributed Consensus: Raft & Paxos

Understand State Machine Replication (SMR), Raft leader election, randomized election timeouts, log replication safety invariants, and joint consensus cluster membership changes.

Primary specification: In Search of an Understandable Consensus Algorithm - Raft (Ongaro & Ousterhout)

Written & reviewed by Yash Patel 30 minBaseline 2026.1 Curated Blueprint· Verified 2026-09-14

Lesson 20 of 33

1d streak

The Architectural Question

When master-slave clusters encounter network partitions, both sides may claim leadership (split-brain), accepting conflicting writes and destroying data consistency.

Core Engineering Objectives

  • State Machine Replication (SMR) concept and consensus role in distributed coordinators (etcd, ZooKeeper)
  • Raft leader election mechanics: heartbeats, terms, randomized timeouts, and split-vote mitigation
  • Raft log replication: commit index, leader completeness property, and cluster membership changes

Step 1: The Core Problem: Consensus Under Asynchronous Network Failure

In a distributed system built from commodity hardware, machines crash, reboot, experience non-deterministic garbage collection pauses, and communicate over asynchronous networks prone to packet loss, reordering, and arbitrary transmission delays. Despite these adverse conditions, modern cloud infrastructure requires clusters of independent machines to act as a single, highly available, fault-tolerant entity.

The Replicated State Machine (RSM) Problem

A Replicated State Machine operates on a simple principle: if identical deterministic state machines start in the same initial state and execute the exact same sequence of state transitions in the exact same order, they will arrive at the exact same state and produce identical outputs. The fundamental challenge of distributed consensus is guaranteeing that all non-faulty nodes agree on the exact sequence of entries in an append-only transaction log, even when nodes crash and network partitions bifurcate the cluster.

Without a mathematically proven consensus protocol, distributed databases inevitably succumb to split-brain syndromes, lost updates, and state divergence. If two nodes in different network partitions simultaneously believe they are the authoritative primary, both accept conflicting client writes, destroying data integrity beyond repair.

Step 2: Requirements, Quorum Mathematics, and Theoretical Bounds

Consensus protocols must guarantee two fundamental properties defined in distributed systems theory: Safety (nothing bad happens) and Liveness (something good eventually happens).

Safety (Never Violate Invariants)

The protocol must never return an incorrect result, permit two leaders in the same term, or apply conflicting log entries at the same index. Even under catastrophic network partitions or arbitrary message delays, a consensus protocol must prefer total unavailability over data corruption.

Liveness (Eventual Progress)

As long as a strict majority of nodes are operational and can communicate with each other over bounded network delays, the cluster must make forward progress, elect a leader, and commit client commands.

Quorum Majority Rule

In a cluster of N nodes, any decision (leader election or log commitment) requires an affirmative vote from a strict majority quorum Q = floor(N / 2) + 1. Any two majorities must overlap by at least one node, ensuring that the latest committed state is always visible to a newly elected quorum.

FLP Impossibility Theorem (Fischer, Lynch, Paterson, 1985)

In a fully asynchronous network, no deterministic consensus protocol can guarantee both safety and liveness if even a single process is subject to unannounced crash failure. Practical consensus algorithms (Raft, Multi-Paxos) circumvent FLP by relying on partial synchrony: weak timing assumptions, randomized timeouts, and heartbeat intervals.

Cluster Size (N)Fault Tolerance (F = floor((N-1)/2))Quorum Size (Q = floor(N/2)+1)Resilience Characterization
1 node0 nodes1 nodeSingle Point of Failure (SPOF); zero fault tolerance
2 nodes0 nodes2 nodesWorst topology: losing 1 node halts the entire cluster (Q=2)
3 nodes1 node2 nodesTolerates 1 crash; ideal for lightweight coordination
4 nodes1 node3 nodesSub-optimal: tolerates only 1 crash but requires 3 nodes for quorum
5 nodes2 nodes3 nodesIndustry standard: tolerates 2 concurrent crashes or rolling deploys
7 nodes3 nodes4 nodesHigh resilience; WAN coordination; increased write latency overhead

The Even-Node Anti-Pattern

Notice that a 4-node cluster provides the exact same fault tolerance (1 node failure) as a 3-node cluster, but requires 3 votes instead of 2 for a quorum. Running an even number of consensus nodes adds hardware cost and network overhead while offering zero additional failure tolerance.

Step 3: Naive Architecture & Why It Catastrophically Collapses

The most frequent failure in amateur distributed architectures is attempting to build consensus with a naive Active-Passive primary/secondary heartbeat mechanism without a strict quorum.

Architectural Boundary · Do Not Confuse

Concept A

Naive Heartbeat Failover (2 Nodes)

Concept B

Quorum Consensus (Raft / Paxos)

In a 2-node cluster, Node A is Primary and Node B is Standby. If an inter-switch network cable drops, Node B stops receiving heartbeats and assumes Node A has died. Node B promotes itself to Primary. Meanwhile, Node A is healthy and continues accepting writes from clients on its side of the network. Both nodes mutate local state independently: split-brain disaster. In Raft, a minority node cannot achieve quorum (Q=2 required) and immediately rejects writes.

The Uncoordinated Disk Flush Trap

Even with 3 nodes, a naive implementation that acknowledges log entries to clients before guaranteeing durable fsync to persistent non-volatile storage will suffer silent corruption. If Node 1 receives a write, buffers it in the OS page cache, replies to the client, and suddenly loses power before disk flush, on reboot Node 1 will vote for a candidate with an older log, causing previously acknowledged user data to vanish into thin air.

Step 4: Evolved Production Architecture: Raft Replicated State Machine

Designed by Diego Ongaro and John Ousterhout at Stanford University (2014), Raft decomposes consensus into three independent, formal sub-problems: Leader Election, Log Replication, and Safety Invariants. Raft enforces a strong leader paradigm: log entries flow strictly from the elected leader to followers.

snippet.typescripttypescript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
// PRODUCTION RAFT REPLICATED STATE MACHINE ENGINE CORE
export type NodeRole = 'FOLLOWER' | 'CANDIDATE' | 'LEADER';

export interface LogEntry<T> {
  index: number;
  term: number;
  command: T;
}

export interface RequestVoteArgs {
  term: number;           // Candidate's term
  candidateId: string;    // Candidate requesting vote
  lastLogIndex: number;   // Index of candidate's last log entry
  lastLogTerm: number;    // Term of candidate's last log entry
}

export interface RequestVoteReply {
  term: number;           // CurrentTerm, for candidate to update itself
  voteGranted: boolean;   // True means candidate received vote
}

export interface AppendEntriesArgs<T> {
  term: number;           // Leader's term
  leaderId: string;       // So follower can redirect clients
  prevLogIndex: number;   // Index of log entry immediately preceding new ones
  prevLogTerm: number;    // Term of prevLogIndex entry
  entries: LogEntry<T>[]; // Log entries to store (empty for heartbeat)
  leaderCommit: number;   // Leader's commitIndex
}

export interface AppendEntriesReply {
  term: number;           // CurrentTerm, for leader to update itself
  success: boolean;       // True if follower contained entry matching prevLogIndex/prevLogTerm
  matchIndex: number;     // Follower's actual log index for fast backoff
}

export class RaftNode<T> {
  public id: string;
  public peers: string[];
  public currentTerm: number = 0;
  public votedFor: string | null = null;
  public role: NodeRole = 'FOLLOWER';
  public log: LogEntry<T>[] = [];
  public commitIndex: number = 0;
  public lastApplied: number = 0;

  // Leader-only state (reinitialized after election)
  public nextIndex: Map<string, number> = new Map();
  public matchIndex: Map<string, number> = new Map();

  constructor(id: string, peers: string[]) {
    this.id = id;
    this.peers = peers;
  }

  // Raft Safety Invariant: RequestVote RPC handler
  public handleRequestVote(args: RequestVoteArgs): RequestVoteReply {
    // 1. Rule: If term < currentTerm, reject vote immediately
    if (args.term < this.currentTerm) {
      return { term: this.currentTerm, voteGranted: false };
    }

    // If newer term discovered, convert to follower
    if (args.term > this.currentTerm) {
      this.currentTerm = args.term;
      this.role = 'FOLLOWER';
      this.votedFor = null;
    }

    // 2. Election Safety: Candidate log must be at least as up-to-date as receiver's log
    const lastLog = this.log[this.log.length - 1];
    const myLastTerm = lastLog ? lastLog.term : 0;
    const myLastIndex = lastLog ? lastLog.index : 0;

    const logIsUpToDate =
      args.lastLogTerm > myLastTerm ||
      (args.lastLogTerm === myLastTerm && args.lastLogIndex >= myLastIndex);

    const canVote = (this.votedFor === null || this.votedFor === args.candidateId) && logIsUpToDate;

    if (canVote) {
      this.votedFor = args.candidateId;
      return { term: this.currentTerm, voteGranted: true };
    }

    return { term: this.currentTerm, voteGranted: false };
  }

  // Raft Safety Invariant: AppendEntries RPC handler (Log Replication + Heartbeat)
  public handleAppendEntries(args: AppendEntriesArgs<T>): AppendEntriesReply {
    // 1. Reply false if term < currentTerm
    if (args.term < this.currentTerm) {
      return { term: this.currentTerm, success: false, matchIndex: this.commitIndex };
    }

    // Recognize valid leader
    if (args.term > this.currentTerm || this.role === 'CANDIDATE') {
      this.currentTerm = args.term;
      this.role = 'FOLLOWER';
      this.votedFor = null;
    }

    // 2. Log Matching Invariant: Check if entry exists at prevLogIndex with prevLogTerm
    if (args.prevLogIndex > 0) {
      const prevEntry = this.log[args.prevLogIndex - 1];
      if (!prevEntry || prevEntry.term !== args.prevLogTerm) {
        return { term: this.currentTerm, success: false, matchIndex: this.log.length };
      }
    }

    // 3. Truncate any conflicting entries and append new entries
    for (let i = 0; i < args.entries.length; i++) {
      const entry = args.entries[i];
      const targetIdx = args.prevLogIndex + i;
      if (this.log[targetIdx] && this.log[targetIdx].term !== entry.term) {
        this.log = this.log.slice(0, targetIdx); // Truncate conflicting suffix
      }
      if (!this.log[targetIdx]) {
        this.log.push(entry);
      }
    }

    // 4. Update commitIndex: min(leaderCommit, index of last new entry)
    if (args.leaderCommit > this.commitIndex) {
      this.commitIndex = Math.min(args.leaderCommit, this.log.length);
    }

    return { term: this.currentTerm, success: true, matchIndex: this.log.length };
  }
}

Use it when

Deploy Raft (e.g., etcd, TiKV, HashiCorp Consul) when your architecture requires linearizable state machine replication, automatic leader election, and strong read-after-write consistency.

Reconsider when

Use Dynamo-style gossip protocols and CRDTs (e.g., Cassandra, ScyllaDB) when your workload demands multi-region high write availability and can tolerate eventual consistency.

Step 5: Step-by-Step Message Flow Trace: Election & Quorum Commit

  1. 01Phase 1: Election Timeout Expiration. Follower Node 2 stops receiving heartbeats from Leader 1 for a randomized duration between 150ms and 300ms. Node 2 transitions to CANDIDATE, increments currentTerm (e.g., Term 4 -> 5), votes for itself, and broadcasts RequestVote RPCs to all peers.
  2. 02Phase 2: Quorum Majority Vote. Node 3 and Node 4 inspect Node 2’s RequestVote payload. Since Node 2’s log is at least as up-to-date as theirs and neither has voted in Term 5, both reply with voteGranted: true. Having secured 3 votes out of 5 (a strict majority quorum), Node 2 crowns itself LEADER.
  3. 03Phase 3: Immediate Heartbeat Suppression. Leader 2 immediately broadcasts an empty AppendEntries RPC (heartbeat) to all nodes. Upon receiving this message, Node 1 and Node 5 recognize Term 5 and revert to FOLLOWER state, resetting their election timers.
  4. 04Phase 4: Client Command Ingestion. A client submits write SET user:100 "Yash" to Leader 2. Leader 2 appends the command to its local log as {index: 42, term: 5, command: ...} without executing it on the state machine yet.
  5. 05Phase 5: AppendEntries Broadcast. Leader 2 sends AppendEntries containing the new entry to all 4 followers. Followers verify that prevLogIndex: 41 and prevLogTerm: 4 match their local logs, append entry 42 to their WALs, and reply success: true.
  6. 06Phase 6: Quorum Commitment & State Machine Execution. Once Leader 2 receives positive ACKs from a majority (itself + Follower 3 + Follower 4), the entry is officially COMMITTED. Leader 2 applies entry 42 to its local state machine, increments commitIndex to 42, and returns HTTP 200 OK to the client.
  7. 07Phase 7: Follower Commit Piggybacking. In the next periodic heartbeat, Leader 2 sets leaderCommit: 42. Followers observe that leaderCommit > commitIndex and execute entry 42 on their local state machines in identical deterministic order.

Linearizable Reads: The ReadIndex & Lease Read Optimization

To read data without stale anomalies, a naive Raft implementation would write every read query to the log as an entry. Production systems optimize this: ReadIndex queries the leader, which checks its current commitIndex, sends a minimal heartbeat to confirm it is still the legitimate leader to a quorum, and returns the state machine value once lastApplied >= commitIndex. Alternatively, CockroachDB and etcd use Leader Leases: if bounded local clock drift guarantees the lease has not expired, reads bypass quorum round-trips entirely!

Step 6: Production War Stories: Kubernetes, CockroachDB, and TiKV

Kubernetes etcd: The Disk Fsync Stalling Outage

In large-scale Kubernetes clusters running 5,000+ nodes, etcd stores the entire cluster specification. During high pod churn, etcd WAL writes must execute synchronous disk flushes (fdatasync). In a famous 2020 cloud outage, slow underlying cloud block storage caused fdatasync latency to spike from 2ms to 650ms. Because the Raft heartbeat interval was 100ms and the election timeout was 1000ms, the etcd leader blocked inside the disk flush thread and failed to emit heartbeats. Followers timed out, declared election, and continuously dethroned leaders, causing a cluster-wide leader election storm that paralyzed the Kubernetes API server fleet.

CockroachDB: Multi-Raft Range Storms

Unlike etcd which runs a single Raft consensus group for the entire database, CockroachDB implements Multi-Raft: data is divided into 64MB key ranges, with each range managed by its own independent Raft consensus group. In early versions, a cluster with 100,000 ranges was broadcasting millions of Raft heartbeats per second across the network, saturating gigabit switches. CockroachDB solved this with Range Leases and Quiescent Ranges: inactive ranges enter hibernation and stop emitting heartbeats until a client read or write activates the range.

Kyle Kingsbury (Jepsen): The Asymmetric Partition Postmortem

In Jepsen testing of distributed consensus systems, Kyle Kingsbury discovered the "Asymmetric Partition" anomaly: Node 1 can send packets to Node 2 and Node 3, but cannot receive packets from Node 3. In early Raft implementations without the Pre-Vote protocol, an isolated candidate with a partitioned inbound connection would repeatedly time out, increment its term number, and spam peers with vote requests. When the partition healed, its artificially inflated term would force the true working leader to step down, causing unnecessary service disruptions.

Step 7: Hard Trade-Offs & Architectural Decision Matrix

Consensus ProtocolPioneered By / ImplementationsLeader ModelUnderstandability & AuditabilityWAN ThroughputProduction Trade-off
Basic PaxosLeslie Lamport (1998)Leaderless / Multi-proposerNotoriously difficult; formal proof onlyLow (dual round-trips for single write)Theoretical foundation; rarely implemented directly in production
Multi-PaxosChubby, Spanner, MegastoreStable leader (proposer lease)Complex state machine & recovery edge casesHigh (pipelined commits)Extremely fast in production, but notoriously hard to implement bug-free
Raftetcd, TiKV, CockroachDB, ConsulStrong leader (log flows 1-way)High (explicit sub-problem decomposition)High (pipelined with batching)Slightly more restrictive leader model than Paxos, but vastly safer to maintain
Zab (ZooKeeper)Apache ZooKeeperPrimary-Backup with epoch leaderModerate (tailored to tree state)High (atomic broadcast FIFO)Tightly coupled to ZooKeeper hierarchical data model; difficult to embed

Architectural Boundary · Do Not Confuse

Concept A

Single-Server Membership Change

Concept B

Joint Consensus ($C_{old,new}$)

In Raft, changing cluster membership dynamically is perilous. If multiple servers are added or removed concurrently, disjoint majorities can elect two conflicting leaders simultaneously. Single-Server changes add or remove exactly one node at a time; because any two majorities of N and N+1 overlap, disjoint majorities are mathematically impossible. Joint Consensus uses a two-phase transition where decisions require dual majorities from both the old configuration and the new configuration simultaneously.

Step 8: Failure Modes, Edge Cases & Cascading Outages

  1. 01The Disruptive Partitioned Node: A network partition isolates Node 5 from the leader. Node 5 times out, increments its term to 99, and attempts election. When the partition heals, Node 5 broadcasts a message with term 99. The healthy leader sees term 99, immediately relinquishes leadership, and triggers an election across the entire cluster. Mitigation: The Pre-Vote Protocol. A node must conduct an informal pre-vote round without incrementing its term. If it cannot get pre-votes from a majority, it cannot initiate an election.
  2. 02The Figure 8 Anomaly (Committing Previous-Term Entries): A leader cannot safely commit a log entry from a previous term simply by counting replicas. If an old entry is replicated to a majority of nodes and the leader crashes before writing an entry in its current term, a newly elected leader can legally overwrite that entry! Rule: A Raft leader must never commit an entry from a previous term by counting replicas; it must commit an entry from its CURRENT term, which indirectly commits all preceding entries.
  3. 03Disk Full / Memory Exhaustion (Log Compaction): An append-only log growing indefinitely will exhaust memory and storage. Raft uses Snapshots: the state machine dumps its point-in-time state to disk, and the log is discarded up to that index. If a slow follower lags behind the snapshot point, the leader transmits the entire snapshot via InstallSnapshot RPC.

Step 9: Interviewer Deep-Dive Gauntlet (Staff/Principal Level)

Deep-Dive Q1: Why can a Raft leader count replicas to commit current-term entries, but not older-term entries?

Model Answer: In standard Raft, if a node is partitioned away from the cluster (e.g., partitioned in an isolated rack), its election timer repeatedly expires. On each timeout, it increments its local term counter (e.g., from Term 5 to Term 45) and sends RequestVote RPCs that no one receives. When the network partition is resolved, this node reconnects to the healthy cluster and broadcasts a message bearing Term 45. Because Raft rules state that any node seeing a higher term must immediately revert to Follower, the current legitimate leader steps down, aborting in-flight transactions and forcing an election storm. The Pre-Vote protocol prevents this by adding an informal reconnaissance phase: before incrementing its term, a candidate sends a PreVote RPC. Peers reply affirmatively only if they have not heard from a valid leader within their election timeout. Because the healthy majority is still receiving heartbeats, they reject the PreVote, preventing the isolated node from ever incrementing its term.

Deep-Dive Q2: Why is a Raft leader prohibited from directly committing a log entry from a previous term by counting replicas (the famous "Figure 8" problem)?

Model Answer: Described in Section 5.4.2 of the Raft paper, consider this scenario: Leader A in Term 2 replicates an entry at index 2 to two servers, then crashes before committing it. In Term 3, Leader B is elected (by servers without this entry) and writes a new entry at index 2, but crashes before replicating it. In Term 4, Server A is re-elected and resumes replication of its Term 2 entry to a majority of servers. Even though this Term 2 entry now resides on a majority of nodes, it is NOT safe to consider it committed. If Server A crashes again, Server B can be elected leader for Term 5 (because it has the Term 3 entry, which has a higher term than Term 2 entries on other servers) and will overwrite the Term 2 entry on all servers! To avoid this hazard, Raft dictates that leaders never commit log entries from previous terms by counting replicas; they commit entries only from their CURRENT term. By committing an entry from the current term, the Leader Completeness property guarantees that all previous uncommitted entries are transitively committed.

Deep-Dive Q3: How do ReadIndex and Lease Reads achieve linearizable reads in Raft without incurring disk write overhead?

Model Answer: Writing read queries into the Raft log guarantees linearizability, but cripples read throughput because every read requires disk fsync on a quorum. ReadIndex eliminates the disk write: 1) The leader records its current commitIndex as readIndex. 2) The leader sends a minimal round of heartbeat messages to a quorum to confirm that it has not been silently deposed by a network partition. 3) The leader waits for its state machine lastApplied index to reach at least readIndex (serving any pending writes). 4) The leader executes the read against the in-memory state machine and replies to the client. Lease Reads optimize this further: the leader acquires a time-bounded lease from followers. As long as the local clock drift is bounded and the lease duration has not elapsed, the leader knows it cannot be deposed, serving reads with ZERO network round-trips and ZERO disk I/O.

Deep-Dive Q4: Explain why single-server cluster membership changes are inherently safe, whereas naive multi-server configuration changes trigger split-brain.

Model Answer: In a cluster configuration change, we transition from configuration Cold to Cnew. If multiple servers are added or removed concurrently (e.g. replacing a 3-node cluster {1,2,3} with {4,5,6}), there exists a time window where some servers know only Cold while others know only Cnew. Cold has majority {1,2} and Cnew has majority {4,5}; these two majorities do not overlap, so the cluster can simultaneously elect two conflicting leaders! Raft solves this in two ways: 1) Joint Consensus (Cold,new), where configuration changes require overlapping majorities from both Cold and Cnew simultaneously. 2) Single-server membership change (the modern standard): nodes are added or removed one by one. In any transition between N nodes and N+1 nodes (or N-1 nodes), any majority in the new configuration must overlap with any majority in the old configuration. Because disjoint majorities cannot form, split-brain is mathematically precluded.

Step 10: Active Recall & Verification Checkpoint

Active Recall Checkpoint

In a 5-node Raft cluster, a network partition isolates 2 nodes from the remaining 3 nodes. What happens to write requests submitted to the 2-node partition vs. the 3-node partition?

Hint: Calculate the quorum majority requirement for a 5-node cluster.

Show answer & explanation

The quorum requirement for 5 nodes is Q = floor(5/2) + 1 = 3 nodes. The 2-node partition cannot form a quorum; any writes submitted to it will block or be rejected, preserving Safety. The 3-node partition contains a strict majority, can elect a leader if needed, reach quorum ACKs for AppendEntries, and commit writes normally, preserving Liveness.

Active Recall Checkpoint

What is the Leader Completeness property in Raft, and why is it essential for State Machine Replication?

Hint: If an entry is committed in term T, where must it exist in future terms?

Show answer & explanation

The Leader Completeness property states that if a log entry is committed in a given term, that entry will be present in the logs of the leaders for all higher-numbered terms. This ensures that a newly elected leader never overwrites or truncates a committed entry, guaranteeing that state machine transitions are permanent and immutable.

Production Postmortem & Verification

Production Incident Postmortem

  1. Trigger / Setup: A 3-node database cluster loses network connectivity between Node 1 and Nodes 2 & 3 without quorum enforcement.
  2. Production Symptom: Node 1 continues accepting writes while Nodes 2 & 3 elect a new leader and accept conflicting writes (Split-Brain).
  3. Root Cause & Fix: Consensus requires a strict majority quorum ((N/2) + 1); minority partitions must immediately step down and reject writes.

Acceptance Criteria & Outcomes

  • Explain the step-by-step Raft leader election process and how randomized timeouts prevent split-vote deadlocks.
  • Prove the Raft Leader Completeness invariant: committed entries are guaranteed present in all future leaders.
  • Trace client read and write paths through a quorum consensus state machine.

End-of-Chapter Synthesis & Key Takeaway

Distributed Consensus: Raft & Paxos is built around one fundamental rule: Establish clear responsibility and explicit trade-offs.. Before adding components, establish explicit quantitative SLAs for latency, throughput, and consistency. Choose the simplest architecture that meets stated constraints.

Architectural & Technical Interview Review

Core FAANG Interview Questions

3 Questions
Q01
How do you justify introducing Distributed Consensus: Raft & Paxos into a high-scale production architecture?
Click to reveal model answer →Hide answer ↑Requirements & Design Brief

Introduce Distributed Consensus: Raft & Paxos only when measured scale, data size, or failure boundaries exceed single-node capacity. State the exact bottleneck, evaluate operational overhead, and verify that the trade-offs justify the added moving parts.

Source section: Requirements & Design Brief
Q02
What is the primary trade-off and hidden operational cost of Distributed Consensus: Raft & Paxos?
Click to reveal model answer →Hide answer ↑Architecture Trade-offs

It improves scalability and availability, but introduces network hops, serialization overhead, and eventual consistency or coordination complexity.

Source section: Architecture Trade-offs
Q03
How do you handle catastrophic failure or dependency degradation in Distributed Consensus: Raft & Paxos?
Click to reveal model answer →Hide answer ↑Failure Review & Recovery

Implement connect/read timeouts, exponential backoff with jitter, and fallback defaults so downstream degradations do not cause cascading outages.

Source section: Failure Review & Recovery

Production Pitfalls & Anti-Patterns

Critical Failure Anti-Patterns

!

Premature Distributed Complexity

Scattering distributed components before identifying concrete write, read, or storage limits creates accidental points of failure and operational debt.

Production Mitigation: Start with the simplest modular design that satisfies requirements, evolving it only as measured constraints dictate.

!

Unbounded Retries and Cascading Storms

When a dependency degrades, immediate synchronized retries multiply downstream traffic, turning a minor slowdown into a total fleet outage.

Production Mitigation: Enforce strict retry budgets, exponential backoff, and full randomized jitter on all client calls.

!

Neglecting Observability and Metrics

Deploying architecture without distributed tracing or p99 tail latency alerts leaves engineers blind when partial partitions occur.

Production Mitigation: Export metrics for request rate, error rate, p95/p99 duration, and saturation at every service boundary.

Primary Documentation & Specifications

In Search of an Understandable Consensus Algorithm - Raft (Ongaro & Ousterhout)
Previous LessonTime, Clocks & Ordering in Distributed SystemsDistributed MechanicsNext LessonDistributed Locking & Monotonic FencingDistributed Mechanics

In this chapter

01Architectural Question02Step 1: The Core Problem: Consensus Under Asynchronous Network Failure03Step 2: Requirements, Quorum Mathematics, and Theoretical Bounds04Step 3: Naive Architecture & Why It Catastrophically Collapses05Step 4: Evolved Production Architecture: Raft Replicated State Machine06Step 5: Step-by-Step Message Flow Trace: Election & Quorum Commit07Step 6: Production War Stories: Kubernetes, CockroachDB, and TiKV08Step 7: Hard Trade-Offs & Architectural Decision Matrix09Step 8: Failure Modes, Edge Cases & Cascading Outages10Step 9: Interviewer Deep-Dive Gauntlet (Staff/Principal Level)11Step 10: Active Recall & Verification Checkpoint12Production Verification13Synthesis & Review14Primary References

Chapter promise

State Machine Replication (SMR) concept and consensus role in distributed coordinators (etcd, ZooKeeper)