I am staring at a workload that is zillions upon zillions of very tiny packets, and each one is important.  They've got to get there fast.  As fast as possible.  Nagle:  you are not welcome here.  I am seeing some seemingly random jitter, and it's only on this one system.  <confused>

I need to take apart this stack piece by piece, and test each in isolation.   Let's start at the lowest level possible.  RHEL6 includes a kernel module called pktgen (modprobe pktgen).  This module allows you to create network packets, specify it's attributes and send them at the fastest possible rate with the least overhead.

Using pktgen, I was able to achieve over 3.3M packets per second on a 10GB Solarflare NIC.  These packets do not have any protocol TCP/UDP packet processing overhead.  You can watch the receivers netstat/IP counters, though.

Since these are synthetic packets, you have to give pktgen some basic information in order for the packets to be constructed with enough info to get there they're going.  Things like destination IP/MAC, the number of packets  and their size.  I tested tiny packets, 64bytes (because that's what this workload needs).  I also tested jumbo frames just to be sure I was doing it right.

This brings up a habit of mine worth mentioning; purposely mis-tuning your environment to validate your settings.  A sound practice!
To get to 3.3Mpps, I only had to make one key change.  Use a 10x factor for  clone_skb.  Anything less than 10 lead to fewer packets (a value of zero halved the pps throughput as compared to 10).  Anything more than 10 had no performance benefit, so I'm sticking with 10 for now.


 Continue reading "Generating arbitrary network packets using the pktgen kernel module" →