How to implement distributed consensus in Rust?

Distributed consensus in Rust means guaranteeing that a set of processes agree on an ordered log of values despite failures. Practical implementation in Rust is dominated by one algorithm: Raft, due to its understandability and straightforward leader-based replication.

The most reliable path is not to write the protocol from scratch, but to integrate a mature crate with your application. The primary options are:

• openraft – actively maintained fork of raft-rs; modular, supports custom storage, networking, and hardening.
• raft-rs – port of the etcd Raft engine; stable but less flexible.
• async-raft – lightweight but largely superseded by the others.

Writing / embedding these libraries reduces your work to implementing four traits: RaftTypeConfig, RaftLogStorage, RaftStateMachine, and RaftNetwork. That still requires you to supply a real network and storage backend, usually also called “implementing distributed consensus” in applied terms.

A production-grade custom Raft implementation in Rust would require:

1. Persistent core state (current term, voted-for, log entries) stored with crash consistency (fsync before any response, using sled, rocksdb, or a manual append-only WAL).
2. Volatile protocol state: commit index, last-applied, role (Follower, Candidate, Leader).
3. An actor model — one async task per consensus node. tokio is standard for timers and message demultiplexing.
4. Serializable RPC messages via serde: RequestVote{RpcRequest, RpcResponse} and AppendEntries{RpcRequest, RpcResponse} (etcd naming), plus Snapshot messages.
5. A network transport: tokio over TCP/QUIC or a message broker, presenting a clean SendRequest/HandleResponse interface. The message direction distinguishes client requests from internal RPCs.
6. Timeout machinery: randomized election timeout (tokio::time::sleep within select!) and heartbeat interval. Precisely:
   • follower waits for AppendEntries,
   • candidate waits either for quorum of votes or for timeout,
   • leader heartbeats followers at fixed intervals.
7. Log replication and application: leader appends entry, replicates to majority, commits, then applies to the state machine via an external command interface.
8. Safety checks: term monotonicity; only one term voting decision; update voted_for before granting vote; commit index only advances if the log is up-to-date with the leader.

An illustrative core skeleton (non-production but structurally correct):

use serde::{Serialize, Deserialize};
use tokio::sync::mpsc;

#[derive(Clone, Debug, PartialEq)]
enum Role { Follower, Candidate, Leader }

#[derive(Clone, Debug, Serialize, Deserialize)]
struct LogEntry { term: u64, command: Vec<u8> }

struct ConsensusNode {
    id: u64,
    peers: Vec<u64>,
    role: Role,
    persistent: PersistentState,      // term, voted_for, log
    commit_index: u64,
    next_index: HashMap<u64, u64>,    // per-peer
    match_index: HashMap<u64, u64>,
    state_machine: Option<Box<dyn StateMachine>>,
    events: mpsc::UnboundedSender<Event>,
}

impl ConsensusNode {
    async fn run(&mut self) {
        // loop receives from network channel,
        // handles RequestVote / AppendEntries,
        // fires timers for elections and heartbeats.
    }
}

What is often underestimated is testing consensus safety. A minimal correct implementation must also pass:

• single-node log replication,
• leader election under partitions,
• no log loss on leader crash,
• duplicate commands from retries.

Use loom for concurrency stress-testing and deterministic testing in tokio::time to simulate latency and network partitioning.

For systems aiming at non-Byzantine failures (crash, partition), Raft is sufficient. If nodes are suspected of actively malicious behavior (forking, lying), switch to a Byzantine fault-tolerant algorithm such as PBFT or Tendermint, for which tendermint-rs exists but is significantly more complex.

Final recommendation: start with openraft before building your own. The library already handles the immutable core invariants of Raft — terms, log matching, quorum math — and Rust’s type system lets you validate your own state transitions compile-time without sacrificing performance.