Distributed Broadcast Primitives

A C++17 implementation of layered broadcast abstractions for distributed systems, built from the ground up over UDP.

Each layer provides strictly stronger guarantees than the one below it:

+-----------------------------------+
|   FIFO Reliable Broadcast (FRB)   |  Per-sender total order
+-----------------------------------+
|  Uniform Reliable Broadcast (URB) |  Majority-ACK before delivery
+-----------------------------------+
|   Best-Effort Broadcast (BEB)     |  Send to all peers
+-----------------------------------+
|        Perfect Links (PP2P)       |  Retransmit + dedup over UDP
+-----------------------------------+
|           UDP Socket              |  Unreliable datagrams
+-----------------------------------+

How it works

Perfect Links wrap raw UDP with sequence-numbered retransmission and ACK-based deduplication — if the receiver is alive, every message is delivered exactly once.

Best-Effort Broadcast sends a message to every peer via perfect links. Delivery is guaranteed only if the sender doesn't crash.

Uniform Reliable Broadcast adds crash tolerance: a message is delivered only after a majority of processes have acknowledged it. Even if the original sender crashes, any delivered message will eventually be delivered by all correct processes.

FIFO Reliable Broadcast buffers URB-delivered messages and releases them in sequence-number order per sender, preserving the broadcast order.

Project structure

src/
├── include/
│   ├── address.hpp        # Network address type with hashing
│   ├── socket.hpp         # UDP socket wrapper
│   ├── perfectlink.hpp    # Reliable point-to-point links
│   ├── beb.hpp            # Best-effort broadcast
│   ├── urb.hpp            # Uniform reliable broadcast
│   ├── frb.hpp            # FIFO reliable broadcast
│   ├── parser.hpp         # CLI argument parser
│   └── util.hpp           # Serialization helpers (pack/unpack)
└── src/
    ├── main.cpp
    ├── socket.cpp
    ├── perfectlink.cpp
    ├── beb.cpp
    ├── urb.cpp
    ├── frb.cpp
    └── address.cpp

Building

Requires CMake 3.14+ and a C++17 compiler.

./build.sh

This produces the bin/da_proc binary.

Usage

./run.sh --id <PROCESS_ID> --hosts <HOSTS_FILE> --output <OUTPUT_FILE> <CONFIG_FILE>

Hosts file — one line per process:

1 127.0.0.1 11001
2 127.0.0.1 11002
3 127.0.0.1 11003

Config file — number of messages to broadcast:

10

Send SIGINT or SIGTERM to stop a process. It writes its log to the output file:

• b <msg> — broadcast message
• d <sender_id> <msg> — delivered message from sender

Demo

An interactive demo that spawns multiple processes locally and verifies correctness:

python3 demo.py --processes 3 --messages 10

  FIFO Reliable Broadcast Demo
  3 processes x 10 messages

  Architecture
  ==============================================
  FIFO Reliable Broadcast
  Total order per sender
   Uniform Reliable Broadcast
   Majority-ACK delivery
    Best-Effort Broadcast
    Send to all peers
     Perfect Links
     Retransmit until ACK
      UDP Socket
      Unreliable datagram

  Results
  ==================================================

  Process 1  [OK]
    Broadcast: ##########.......... 10 sent
    Recv P1:  ##########.......... 10 delivered
    Recv P2:  ##########.......... 10 delivered
    Recv P3:  ##########.......... 10 delivered

  Summary
  ==================================================
  Broadcast check:   PASS (all 30 messages sent)
  FIFO order check:  PASS (all deliveries in sender order)

Key design decisions

• Layered composition — each broadcast primitive extends the one below via inheritance, mirroring the theoretical model from Cachin, Guerraoui & Rodrigues' Introduction to Reliable and Secure Distributed Programming.
• Zero-copy ACK — acknowledgments reuse the received packet header, flipping a single byte.
• Lock-per-peer — mutexes are partitioned by remote address to reduce contention.
• Stack-allocated receive buffer — the listener uses a fixed 128-byte stack buffer instead of heap allocation per packet.

References

• Cachin, Guerraoui, Rodrigues. Introduction to Reliable and Secure Distributed Programming (2nd ed.), Springer, 2011.