tutorial_dpdk.html

---
title: Tutorials
layout: default
section: tutorials
---
<h1><a href="tutorials.html">Tutorials</a></h1>
<h2>Part 6: Working with DPDK</h2>
<p>This part explains how to work with PcapPlusPlus DPDK APIs</p>
<p>Table of contents:</p>
<ul>
  <li><a href="tutorial_intro.html">Part 1: Introduction and basics</a></li>
  <li><a href="tutorial_pcap_files.html">Part 2: Reading and writing
      pcap files</a></li>
  <li><a href="tutorial_live_capture.html">Part 3: Capturing and
      sending packets</a></li>
  <li><a href="tutorial_packet_parsing.html">Part 4: Packet parsing</a></li>
  <li><a href="tutorial_packet_craft_n_edit.html">Part 5: Packet crafting and editing</a></li>
  <li>Part 6: Working with DPDK
    <ol>
      <li><a href="#intro">Introduction</a></li>
      <li><a href="#setup">Set up DPDK Devices</a> </li>
      <li><a href="#worker_threads">DPDK Worker Threads</a></li>
      <li><a href="#start_worker_threads">Start Worker Threads</a></li>
      <li><a href="#management_n_stats">Management And Statistics Collection</a></li>
      <li><a href="#run_example">Running the example</a></li>
    </ol>
  </li>
</ul>
<h2 id="intro">Introduction</h2>
<p>One of the key advantages of PcapPlusPlus over similar libraries is its extensive support for the Data Plane Development Kit (DPDK). 
  PcapPlusPlus provides an extensive wrapper for DPDK which encapsulates most of its important and commonly used APIs in 
  an easy-to-use C++ way.</p>
<p>This tutorial will go through the fundamentals of using PcapPlusPlus with DPDK. It will demonstrate how to
  build a L2 forwarding (bridge) application that receives packets on one interface and sends them on another interface. It may
  sound simple, but DPDK enables to do that in wire speed (!). This example will demonstrate
  some of the key APIs and concepts in PcapPlusPlus wrapper for DPDK.</p>
<p>This specific example was chosen because it corresponds to a similar example in DPDK documentation called 
  <a href="https://doc.dpdk.org/guides-18.08/sample_app_ug/l2_forward_real_virtual.html">L2 Forwarding</a> which many DPDK
  users are probably familiar with, and that may help in better understanding the code and the idea behind it.</p>
<p>Before starting this tutorial it is highly recommended to have a basic understanding of what DPDK is (you can find a lot of 
  reading material in <a href="http://dpdk.org/">DPDK web-site</a>) and also read the page describing 
  <a href="dpdk-pf_ring.html">PcapPlusPlus support for DPDK</a>.</p>
<p>For further information about the APIs and classes please refer to  the <a href="Documentation/index.html">API
    documentation</a>.</p>
<p>Before diving into the code, let's see how our L2 forwarding (bridge) application will be built:</p>
<img src="resources/L2Fwd.png" width="100%"/>
<p>As you can see, we will use 2 DPDK-controlled NICs, one from each side of the network. We'll also have 2 worker threads.
  The first thread will receive packets on NIC #1 and will send them on NIC #2 and the other thread will receive packets
  on NIC #2 and will send them on NIC #1. Now that we have this basic understanding let's go ahead and build this application!</p>

<h2 id="setup">Set up DPDK Devices</h2>
<p>The first thing any application that uses DPDK should do is initialize DPDK and set up the DPDK interfaces (devices).
  This initialization involves a couple of steps and we'll go through all of them.</p>
<p>The first step is done before running the application. PcapPlusPlus contains a shell script called <code>setup-dpdk.sh</code> which initializes 
  Huge Pages (which are required for DPDK's memory allocation) and DPDK kernel driver which removes kernel control 
  from selected NICs and hand it over to DPDK. You can read more about it in <a href="dpdk-pf_ring.html">PcapPlusPlus support for DPDK</a> page.</p>
<p>The second step is done in the application's code and is a general DPDK initialization phase. It is also described in the
  <a href="dpdk-pf_ring.html">PcapPlusPlus support for DPDK</a> page and contains steps like initialize DPDK internal structures and memory pools,
  initialize packet memory pool, and more. Lets start by writing a general main method and initialize DPDK:</p>  
<pre><code class="language-cpp">#include &lt;vector&gt;
#include &lt;unistd.h&gt;
#include &lt;sstream&gt;
#include "SystemUtils.h"
#include "DpdkDeviceList.h"
#include "TablePrinter.h"
            
#include "WorkerThread.h"

#define MBUF_POOL_SIZE 16*1024-1
#define DEVICE_ID_1 0
#define DEVICE_ID_2 1

int main(int argc, char* argv[])
{
    // Initialize DPDK
    pcpp::CoreMask coreMaskToUse = pcpp::getCoreMaskForAllMachineCores();
    pcpp::DpdkDeviceList::initDpdk(coreMaskToUse, MBUF_POOL_SIZE);

    ....
    ....
}</code>
</pre>
<p>There are couple of steps here:
  <ul>
    <li>Decide of the mbuf pool size. mbufs are DPDK structures for holding packet data. Each mbuf holds one packet data or a portion (segment) of it.
      On application start-up DPDK allocates memory for creating a pool of mbufs that will be used by the application throughout its runtime. That way 
      DPDK avoids the overhead of creating mbufs and allocating memory during application run. Let's decide that the size of this mbuf pool is 16383 mbufs. 
      It is recommended to set a size that is a power of 2 minus 1 (in our case: 16383 = 2^14 - 1)
    </li>
    <li>Decide which CPU cores will take part in running the application. DPDK leverages multi-core architecture to parallelize packet processing. In our case
      we initialize DPDK with all cores available on the machine
    </li>
    <li>Invoke <code class="language-cpp">pcpp::DpdkDeviceList::initDpdk()</code> which runs the initialization</li>
  </ul>
</p>
<p>Now let's find the DPDK interfaces (devices) we'll use to send and receive packets. The class <code class="language-cpp">DpdkDevice</code> encapsulates a DPDK interface.
  The singleton <code class="language-cpp">DpdkDeviceList</code> contains all DPDK devices that are available for us to use:</p>
<pre><code class="language-cpp">// Find DPDK devices
pcpp::DpdkDevice* device1 = pcpp::DpdkDeviceList::getInstance().getDeviceByPort(DEVICE_ID_1);
if (device1 == NULL)
{
    printf("Cannot find device1 with port '%d'\n", DEVICE_ID_1);
    return 1;
}

pcpp::DpdkDevice* device2 = pcpp::DpdkDeviceList::getInstance().getDeviceByPort(DEVICE_ID_2);
if (device2 == NULL)
{
    printf("Cannot find device2 with port '%d'\n", DEVICE_ID_2);
    return 1;
}
  </code>
</pre>
<p>As you can see, we're using the <code class="language-cpp">DpdkDeviceList</code> singleton to get the 2 DPDK devices. 
  The port numbers <i>DEVICE_ID_1</i> (of value 0) and <i>DEVICE_ID_2</i> (of value 1) are determined by DPDK and we should know
  them in advance.</p>
<p>The next step is to open the 2 devices so we can start receiving and sending packets through them. Let's see the code:</p>
<pre><code class="language-cpp">// Open DPDK devices
if (!device1->openMultiQueues(1, 1))
{
    printf("Couldn't open device1 #%d, PMD '%s'\n", device1->getDeviceId(), device1->getPMDName().c_str());
    return 1;
}
      
if (!device2->openMultiQueues(1, 1))
{
    printf("Couldn't open device2 #%d, PMD '%s'\n", device2->getDeviceId(), device2->getPMDName().c_str());
    return 1;
}</code>
</pre>
<p>As you can see we're using a method called <code class="language-cpp">openMultiQueues()</code>. This method opens the device
  with a provided number of RX and TX queues. The number of supported RX and TX queues varies between NICs. You can get
  the number of supported queues by using the following methods: <code class="language-cpp">DpdkDevice::getTotalNumOfRxQueues()</code>
  and <code class="language-cpp">DpdkDevice::getTotalNumOfTxQueues()</code>. The reason for opening more than 1 RX/TX queue is to
  parallelize packet processing over multiple cores where each core is responsible for 1 or more RX/TX queues. On RX, 
  The NIC is responsible to load-balance RX packets to the different queues based on a provided hash function. Doing this
  inside the NIC makes it much faster and offloads processing from CPU cores. This load balancing mechanism 
  is called Receive Side Scaling (RSS) and is also wrapped by PcapPlusPlus, for more details please see RSS configuration in
  <code class="language-cpp">DpdkDevice::DpdkDeviceConfiguration</code>.</p>
<p>In our case we choose the simple case of 1 RX queue and 1 TX queue for each device. That means we'll use 1 thread for each direction.</p>

<h2 id="worker_threads">DPDK Worker Threads</h2>
<p>Now that we finished the DPDK setup and initialization, let's move on to the actual work of capturing and sending packets.
  The way we are going to do that is using DPDK worker threads. We will create 2 worker threads: one for sending packets from device1
  to device2, and the other for sending packets from device2 to device1. Each worker thread will run on a separate CPU core and will
  execute an endless loop that will receive packets from one device and send them to the other.</p>
<p>Worker threads on PcapPlusPlus are instances of a class that inherits <code class="language-cpp">DpdkWorkerThread</code>. Let's write the 
  header file of this class and see how it looks like:</p>
<pre><code class="language-cpp">#pragma once

#include "DpdkDevice.h"
#include "DpdkDeviceList.h"

class L2FwdWorkerThread : public pcpp::DpdkWorkerThread
{
  private:
    pcpp::DpdkDevice* m_RxDevice;
    pcpp::DpdkDevice* m_TxDevice;
    bool m_Stop;
    uint32_t m_CoreId;

  public:
    // c'tor
    L2FwdWorkerThread(pcpp::DpdkDevice* rxDevice, pcpp::DpdkDevice* txDevice);

    // d'tor (does nothing)
    ~L2FwdWorkerThread() { }

    // implement abstract method

    // start running the worker thread
    bool run(uint32_t coreId);

    // ask the worker thread to stop
    void stop();
              
    // get worker thread core ID
    uint32_t getCoreId();
};</code>
</pre>
<p><code class="language-cpp">DpdkWorkerThread</code> is an abstract class that requires inherited classes to implement 3 methods:</p>
<ul>
  <li><code class="language-cpp">run()</code> - start the worker. This method is called when the thread gets invoked and is 
    expected to run throughout the life of the thread. Typically this method will contain an endless loop that runs the logic of the 
    application</li>
  <li><code class="language-cpp">stop()</code> - stop the execution of the worker</li>
  <li><code class="language-cpp">getCoreId()</code> - return the core ID the worker is running on</li>
</ul>
<p>In addition to implementing these method we also have a constructor and an empty destructor. We also save pointers to the
  RX and TX devices of where the worker will read packets from and send packets to. Now let's see the implementation of
  this class's methods:</p>
<pre><code class="language-cpp">#include "WorkerThread.h"


L2FwdWorkerThread::L2FwdWorkerThread(pcpp::DpdkDevice* rxDevice, pcpp::DpdkDevice* txDevice) :
  m_RxDevice(rxDevice), m_TxDevice(txDevice), m_Stop(true), m_CoreId(MAX_NUM_OF_CORES+1)
{
}

bool L2FwdWorkerThread::run(uint32_t coreId)
{
  // Register coreId for this worker
  m_CoreId = coreId;
  m_Stop = false;
              
  // initialize a mbuf packet array of size 64
  pcpp::MBufRawPacket* mbufArr[64] = {};
              
  // endless loop, until asking the thread to stop
  while (!m_Stop)
  {
    // receive packets from RX device
    uint16_t numOfPackets = m_RxDevice->receivePackets(mbufArr, 64, 0);

    if (numOfPackets > 0)
    {
      // send received packet on the TX device
      m_TxDevice->sendPackets(mbufArr, numOfPackets, 0);
    }
  }

  return true;
}
              
void L2FwdWorkerThread::stop()
{
  m_Stop = true;
}
              
uint32_t L2FwdWorkerThread::getCoreId()
{
  return m_CoreId;
}
</code>
</pre>
<p>The constructor is quite straight forward and initializes the private members. Please notice that the initialized value 
  for the core ID is the maximum supported number of cores + 10.</p>
<p>The <code class="language-cpp">stop()</code> and <code class="language-cpp">getCoreId()</code> methods are also quite trivial and 
  self explanatory.</p>
<p>Now let's take a look at the <code class="language-cpp">run()</code> method which contains the L2 forwarding logic. 
  It consists of an endless loop that is interrupted through a flag set by the <code class="language-cpp">stop()</code>
  method (which indicates the thread should stop its execution). Before starting the loop it creates an array of 64
  <code class="language-cpp">MBufRawPacket</code> pointers which will be used to store the received packets. The loop itself
  is very simple: it receives packets from the RX device using <code class="language-cpp">m_RxDevice->receivePackets(mbufArr, 64, 0)</code>.
  The packets are stored in the <code class="language-cpp">MBufRawPacket</code> array. Then it immediately sends those packets to the 
  TX device using <code class="language-cpp">m_TxDevice->sendPackets(mbufArr, numOfPackets, 0)</code>. You may be asking who takes care
  of freeing the packet array and mbufs in each iteration of the loop. Well, this is done automatically by 
  <code class="language-cpp">sendPackets()</code> so we don't have to take care of it ourselves.</p>
<p>This basically summarizes the implementation of the worker thread. In the current application we'll set up 2 worker threads:
  one for receiving packets from DEVICE_ID_1 and send them over DEVICE_ID_2 and another to receiving packets from DEVICE_ID_2
  and send them over DEVICE_ID_1.</p>


<h2 id="start_worker_threads">Start Worker Threads</h2>
<p>Now that we have the worker thread code ready, let's wire everything up and start the application. First, let's create the
  worker thread instances:</p>
<pre><code class="language-cpp">// Create worker threads
std::vector&lt;pcpp::DpdkWorkerThread*&gt; workers;
workers.push_back(new L2FwdWorkerThread(device1, device2));
workers.push_back(new L2FwdWorkerThread(device2, device1));</code>
</pre>
<p>As you can see we give the first worker thread device1 as the RX device and device2 as the TX device, and vice versa for the
  second worker thread. We store pointers to these two instances in a vector.</p>
<p>Next step is to assign cores for these two worker threads to run on. DPDK enforces running each worker in a 
  separate core to maximize performance. We will create a core mask that contains core 1 and core 2, let's see how this code
  looks like:</p>
<pre><code class="language-cpp">// Create core mask - use core 1 and 2 for the two threads
int workersCoreMask = 0;
for (int i = 1; i &lt;= 2; i++)
{
  workersCoreMask = workersCoreMask | (1 &lt;&lt; (i+1));
}</code>
</pre>
<p>As you can see, we basically create the value 0x6 (or 0b110) where we set only the bits who correspond to the cores we want to use (1 and 2)</p>
<p>Now let's start the worker threads:</p>
<pre><code class="language-cpp">// Start capture in async mode
if (!pcpp::DpdkDeviceList::getInstance().startDpdkWorkerThreads(workersCoreMask, workers))
{
	printf("Couldn't start worker threads");
	return 1;
}</code>
</pre>


<h2 id="management_n_stats">Management And Statistics Collection</h2>
<p>Now wer'e at a point where the 2 worker threads are running their endless loops which receives packets on one interface and sends them 
  to the other interface. Practically we're done and the bridge should be working now. But to make the program more complete let's also 
  add a graceful shutdown and user friendly prints to view the RX/TX statistics during application run.</p>
<p>For the graceful shutdown we'll use an utility class in PcapPlusPlus called <code class="language-cpp">ApplicationEventHandler</code> which 
encapsulates user-driven events that may occur during application run, such as process kill (ctrl+c). For using this class we'll need
to add one line at the beginning of our <code class="language-cpp">main()</code> method which registers a callback we'd like to be called when  
ctrl+c is pressed:</p>
<pre><code class="language-cpp">int main(int argc, char* argv[])
{
  // Register the on app close event handler
  pcpp::ApplicationEventHandler::getInstance().onApplicationInterrupted(onApplicationInterrupted, NULL);

  // Initialize DPDK
  pcpp::CoreMask coreMaskToUse = pcpp::getCoreMaskForAllMachineCores();
  pcpp::DpdkDeviceList::initDpdk(coreMaskToUse, MBUF_POOL_SIZE);
  
  .....
  .....</code>
</pre>
<p>Now let's implement this <code class="language-cpp">onApplicationInterrupted</code> callback. It'll have a very simple logic which sets a global flag:</p>
<pre><code class="language-cpp">// Keep running flag
bool keepRunning = true;

void onApplicationInterrupted(void* cookie)
{
  keepRunning = false;
  printf("\nShutting down...\n");
}</code>
</pre>
<p>Now that we have this flag we can set up an endless loop that will run on the main thread and will keep printing statistics until ctrl+c is 
    pressed. Please notice this is not the loop the worker threads are running, this is a different loop that runs on the management core
    (core 0 in our case). Let's dwell on this point a little bit more to better understand how DPDK works: the worker threads are running on 
    cores 1 and 2 and their endless loop consumes 100% of their capacity. This guarantees achieving the best possible 
    performance. However it's a good practice (although not required) to allocate at least one more CPU core for management, meaning 
    tasks that are not in the application's fast-path, such as statistics collection, provide user-interface (CLI or other), health monitoring,
    etc. Usually this management core will be core 0, but you can set up any other core. This management core is also the one running the 
    <code class="language-cpp">main()</code> method. Now let's go back to our application: once we started the worker threads on cores 1 and 2, we would
    like the management core to continuously gather statistics and print them to the user. The way to do that is to set up and endless loop inside 
    the <code class="language-cpp">main()</code> method that will collect and print the stats and will be interrupted when the user presses ctrl+c 
    (and setting the <code class="language-cpp">keepRunning</code> flag). Let's see the implementation:</p>
<pre><code class="language-cpp">#define COLLECT_STATS_EVERY_SEC 2
uint64_t counter = 0;
int statsCounter = 1;
              
// Keep running while flag is on
while (keepRunning)
{
  // Sleep for 1 second
  sleep(1);

  // Print stats every COLLECT_STATS_EVERY_SEC seconds
  if (counter % COLLECT_STATS_EVERY_SEC == 0)
  {
    // Clear screen and move to top left
    const char clr[] = { 27, '[', '2', 'J', '\0' };
    const char topLeft[] = { 27, '[', '1', ';', '1', 'H','\0' };
    printf("%s%s", clr, topLeft);

    printf("\n\nStats #%d\n", statsCounter++);
    printf("==========\n\n");

    // Print stats of traffic going from Device1 to Device2
    printf("\nDevice1->Device2 stats:\n\n");
    printStats(device1, device2);

    // Print stats of traffic going from Device2 to Device1
    printf("\nDevice2->Device1 stats:\n\n");
    printStats(device2, device1);
  }

  counter++;
}</code>
</pre>
<p>As you can see the while loop collects statistics, prints them and then sleeps for 1 second.</p>
<p>Now let's see how to gather network statistics:</p>
<pre><code class="language-cpp">void printStats(pcpp::DpdkDevice* rxDevice, pcpp::DpdkDevice* txDevice)
{
  pcpp::DpdkDevice::DpdkDeviceStats rxStats;
  pcpp::DpdkDevice::DpdkDeviceStats txStats;
  rxDevice->getStatistics(rxStats);
  txDevice->getStatistics(txStats);

  std::vector&lt;std::string&gt; columnNames;
  columnNames.push_back(" ");
  columnNames.push_back("Total Packets");
  columnNames.push_back("Packets/sec");
  columnNames.push_back("Bytes");
  columnNames.push_back("Bits/sec");

  std::vector&lt;int&gt; columnLengths;
  columnLengths.push_back(10);
  columnLengths.push_back(15);
  columnLengths.push_back(15);
  columnLengths.push_back(15);
  columnLengths.push_back(15);

  pcpp::TablePrinter printer(columnNames, columnLengths);

  std::stringstream totalRx;
  totalRx &lt;&lt; "rx" &lt;&lt; "|" &lt;&lt; rxStats.aggregatedRxStats.packets &lt;&lt; "|" &lt;&lt; rxStats.aggregatedRxStats.packetsPerSec &lt;&lt; "|" &lt;&lt; rxStats.aggregatedRxStats.bytes &lt;&lt; "|" &lt;&lt; rxStats.aggregatedRxStats.bytesPerSec*8;
  printer.printRow(totalRx.str(), '|');

  std::stringstream totalTx;
  totalTx &lt;&lt; "tx" &lt;&lt; "|" &lt;&lt; txStats.aggregatedTxStats.packets &lt;&lt; "|" &lt;&lt; txStats.aggregatedTxStats.packetsPerSec &lt;&lt; "|" &lt;&lt; txStats.aggregatedTxStats.bytes &lt;&lt; "|" &lt;&lt; txStats.aggregatedTxStats.bytesPerSec*8;
  printer.printRow(totalTx.str(), '|');
}</code>
</pre>
<p><code class="language-cpp">DpdkDevice</code> exposes the <code class="language-cpp">getStatistics()</code> method for stats collection. Various counters are
  being collected such as the number of packets, amount of data, packet per second, bytes per second, etc. You can view them separately per 
  RX/TX queue, or aggregated per device. It's important to understand that these numbers are only relevant for the timestamp they
  are being collected and therefore this timestamp is also included in the data. You can read more about this in the class documentation.</p>
<p>If we go back to the code above, you can see we're collecting stats for the 2 devices. From one we take RX stats and from the other we
  take TX stats. We are using an utility class in PcapPlusPlus called <code class="language-cpp">TablePrinter</code> to print the numbers nicely in a 
  table format. For the sake of simplicity we are taking only the aggregated RX and TX stats, but of course we can also take and prints RX/TX 
  stats per queue.</p>
<p>We are almost done. One last thing to do is to run the necessary clean ups once the user presses ctrl+c. The only relevant clean-up is 
  to stop the worker threads, let's see the code:</p>
<pre>
  <code class="language-cpp">// Stop worker threads
pcpp::DpdkDeviceList::getInstance().stopDpdkWorkerThreads();

// Exit app with normal exit code
return 0;</code>
</pre>
<p>That's it, we're all set! Now let's run the program and see the output:</p>
<pre><code class="language-shell">Stats #5
==========


Device1->Device2 stats:

--------------------------------------------------------------------------------------
|            | Total Packets   | Packets/sec     | Bytes           | Bits/sec        |
--------------------------------------------------------------------------------------
| rx         | 2850754         | 134607          | 4307240599      | 1627406832      |
| tx         | 2851371         | 132058          | 4296728841      | 1592137536      |
--------------------------------------------------------------------------------------

Device2->Device1 stats:

--------------------------------------------------------------------------------------
|            | Total Packets   | Packets/sec     | Bytes           | Bits/sec        |
--------------------------------------------------------------------------------------
| rx         | 160880          | 3273            | 11261910        | 1833416         |
| tx         | 161001          | 4533            | 10627168        | 2393688         |
--------------------------------------------------------------------------------------
</code>
</pre>
<p>This output is printed every 2 seconds and shows for each direction: the total number of packets received and sent so far, 
  the total number of bytes received and sent so far, packets per second and bps (bits per second)</p>


<h2 id="run_example">Running the example</h2>
<p>All the code that was covered in this tutorial can be found <a href="https://github.com/seladb/PcapPlusPlus/tree/master/Examples/Tutorials/Tutorial-DpdkL2Fwd">here</a>.
  In order to compile and run the code please first download and compile PcapPlusPlus source code or download a pre-compiled version from
  <a href="https://github.com/seladb/PcapPlusPlus/releases/latest">the latest PcapPlusPlus release</a>. When building from source please
  make sure to configure the build for DPDK, as explained <a href="dpdk-pf_ring.html">here</a>. The only platform relevant for this
  tutorial is Linux as DPDK is not supported on other platforms.</p>
<p>After done building PcapPlusPlus and the tutorial and before running the tutorial please run <code class="language-shell">setup-dpdk.sh</code> script to setup
  the necessary runtime parameters for DPDK. More details on this script can be found <a href="dpdk-pf_ring.html">here</a>.</p>
<p>Please note this tutorial needs a special environment to run on, as it needs at least 2 devices connected only through a third device running
  this application. If you need help setting up this environment and you have VirtualBox you can use  
  <a href="https://www.brianlinkletter.com/how-to-use-virtualbox-to-emulate-a-network/">this</a> great tutorial which will walk you through it.</p>
<p>The compiled executable will be inside the tutorial directory 
  (<code class="language-shell">[PcapPlusPlus
    Folder]/Examples/Tutorials/Tutorial-DpdkL2Fwd</code>).</p>
<script>function topFunction() { document.body.scrollTop = 0; document.documentElement.scrollTop = 0; }</script>
<p><a onclick="topFunction()" href="#">Go to top &uarr;</a></p>