Skip to content

modern-stack.md

Latest commit

 

History

History
234 lines (175 loc) · 6.54 KB

modern-stack.md

File metadata and controls

234 lines (175 loc) · 6.54 KB
Raw

Modern E2E Benchmarking Stack

A production-grade CI/CD benchmarking pipeline for network data planes uses a four-tier architecture. This is the approach taken by cloud providers, telcos, and projects like FD.io CSIT.

flowchart TD
    subgraph tier1["Tier 1 — Orchestrator"]
        orch["PyTest / Robot Framework\ndefine pass/fail criteria\n\nAnsible / Terraform\nDUT provisioning"]
    end

    subgraph tier2["Tier 2 — Traffic Generators"]
        stateless["TRex stateless\nPktgen-DPDK\n\nL2/L3 raw throughput\nRFC 2544 NDR/PDR"]
        stateful["TRex ASTF\nFlent\n\nReal TCP connections\nBufferbloat / HTTP"]
    end

    subgraph tier3["Tier 3 — Impairment"]
        impair["Linux netem / tc\nToDD\n\nLatency / jitter / loss\nWAN simulation"]
    end

    subgraph tier4["Tier 4 — Telemetry"]
        prom["Prometheus\nscrapes DUT metrics"]
        graf["Grafana\ncorrelates traffic\nvs CPU / drops"]
        ebpf["eBPF / BCC\nkernel-level tracing\nIRQ / softirq bottlenecks"]
    end

    subgraph dut["Device Under Test"]
        vyos["VyOS router"]
    end

    tier1 -->|"provision + configure DUT"| dut
    tier1 -->|"start test"| tier2
    tier2 -->|"traffic"| tier3
    tier3 -->|"impaired traffic"| dut
    dut -->|"metrics"| tier4
    tier4 -->|"results"| tier1
Loading

Tier 1 — Orchestrator

The orchestrator defines tests, sets pass/fail criteria, provisions the DUT, and collects results.

PyTest

Write test cases with explicit pass/fail criteria:

# tests/test_throughput.py
def test_512byte_throughput(benchmark_result):
    """512-byte throughput must exceed 5 Gbps for production readiness."""
    assert benchmark_result["512"] > 5000, (
        f"512-byte throughput {benchmark_result['512']} Mbps below 5000 Mbps threshold"
    )

def test_128byte_throughput(benchmark_result):
    """128-byte throughput must exceed 1 Gbps."""
    assert benchmark_result["128"] > 1000

Robot Framework

Used by FD.io CSIT — test cases expressed as keyword-driven scripts:

*** Test Cases ***
VyOS 512-Byte NDR Throughput
    [Documentation]    Non-Drop Rate must exceed 5 Gbps at 512-byte
    ${result}=    Run TRex NDR Test    packet_size=512
    Should Be True    ${result.ndr_gbps} > 5.0

Ansible for DUT provisioning

Before each test run, Ansible ensures the DUT is in a known configuration:

# playbooks/provision-dut.yml
- name: Reset DUT to baseline config
  hosts: dut
  tasks:
    - name: Apply forwarding-only config
      vyos.vyos.vyos_config:
        src: "configs/{{ test_profile }}.j2"
        save: true

Tier 2 — Traffic generators

Tool Mode Best for
TRex stateless L2/L3 RFC 2544 NDR/PDR, raw PPS
Pktgen-DPDK L2/L3 Maximum PPS measurement
TRex ASTF L4-L7 Concurrent connections, HTTP rate
Flent L4-L7 Bufferbloat, real TCP behaviour
iperf3 L4-L7 Practical throughput (this repo)

See advanced-tools.md for setup and usage of each.

Tier 3 — Impairment and simulation

Place a Linux impairment node between the traffic generator and the DUT. Use tc netem to inject WAN-like conditions:

# On the impairment node
# Simulate a 100ms WAN link with 0.5% loss
sudo tc qdisc add dev eth1 root netem delay 100ms loss 0.5%

Standard test matrix:

Profile Parameters Use case
LAN no impairment Baseline
Good WAN 20ms delay Typical internet
Poor WAN 100ms delay, 0.5% loss Remote site
Stressed WAN 200ms delay ±50ms jitter, 2% loss Degraded link

Tier 4 — Telemetry and observability

Running traffic without observing the DUT tells you the result but not the cause. Tier 4 answers: why did throughput drop at 128-byte packets?

Prometheus + node_exporter on VyOS

VyOS (Linux) can expose metrics via node_exporter:

# On VyOS (run as a service or in a container)
node_exporter --web.listen-address=:9100 \
  --collector.interrupts \
  --collector.softnet \
  --collector.netdev

Key metrics to watch during benchmarking:

Metric What it reveals
node_cpu_seconds_total{mode="softirq"} Kernel network interrupt CPU spend
node_network_receive_drop_total Interface-level drops
node_softnet_dropped_total Kernel network backlog drops
node_network_receive_packets_total PPS per interface

Grafana dashboard

Correlate traffic rate (from TRex/iperf3) against DUT CPU and drop metrics in real time. The standard pattern:

X-axis: time
Y-axis panel 1: throughput Mbps (from iperf3/TRex)
Y-axis panel 2: DUT CPU % softirq
Y-axis panel 3: DUT interface drops/sec

When throughput plateaus and CPU softirq hits 100%, you've found the kernel interrupt processing limit — the exact scenario the Soucy methodology describes for small packets.

eBPF / BCC tools

For deeper kernel-level tracing on the VyOS (Linux) DUT:

# Trace softirq execution time per CPU
/usr/share/bcc/tools/softirqs -d 10

# Trace network receive path
/usr/share/bcc/tools/tcpretrans

# Show interrupt rate per CPU core
watch -n1 'cat /proc/interrupts | grep eth'

Key insight: If one CPU core handles all NIC interrupts (no RSS/RPS), throughput is limited by a single core. Enabling Receive Side Scaling (RSS) distributes interrupts across all cores and can dramatically improve small-packet throughput.

Bringing it together — CI pipeline for benchmarking

# .github/workflows/nightly-benchmark.yml
name: Nightly Benchmark

on:
  schedule:
    - cron: '0 1 * * *'   # 01:00 UTC — before daily VyOS apply

jobs:
  benchmark:
    runs-on: self-hosted   # must be on management network
    steps:
      - uses: actions/checkout@v4

      - name: Provision DUT (forwarding-only)
        run: ansible-playbook playbooks/provision-dut.yml -e "test_profile=forwarding"

      - name: Setup test nodes
        run: ansible-playbook playbooks/setup.yml

      - name: Run benchmark suite
        run: |
          ansible-playbook playbooks/benchmark.yml \
            -e "save_results=true" \
            -e "result_label='nightly-forwarding-$(date +%F)'"

      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-${{ github.run_id }}
          path: results/latest.md

      - name: Teardown
        if: always()
        run: ansible-playbook playbooks/teardown.yml

This pattern — nightly benchmark after the daily VyOS config apply — catches performance regressions introduced by config changes before they reach production.