LAN Ethernet Maximum Rates, Generation, Capturing & Monitoring
Overview
Designing and managing an IP network requires an in-depth understanding of both the network infrastructure and the performance of devices that are attached, including how packets are handled by each network device. Network computing engineers most often refer to the performance of network devices by using the speed of the interfaces expressed in bits per second (bps). For example, a network device may be described as having a performance of 10 gigabits per second (Gbps). Although this is useful and important information, expressing performance in terms of bps alone does not adequately cover other important network device performance metrics. Determining effective data rates for varying Ethernet packets sizes can provide vital information and a more complete understanding of the characteristics of the network.
It is the intent of this article to measure maximum LAN Ethernet rates values that can be achieved for both Fast Ethernet and Gigabit Ethernet using the TCP/IP or the UDP network protocols. A stepwise approach on how to generate, capture and monitor maximum Ethernet rates will be shown using various tools bundled with the Network Security Toolkit (NST). We will follow many of the methods described in RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" to perform benchmark tests and performance measurements.
A demonstration and discussion on how Linux segmentation off-loading for supported NIC adapters can produce Super Ethernet Frames will be presented. These large Ethernet frames can be captured and decoded by the network protocol analyzer. The use of segmentation off-loading can result in increased network performance and less CPU overhead for network packet processing. Finally, a section on Ethernet flow control (IEEE 802.3x) will be review and its effects during high data transfer rates will be revealed.
Ethernet Maximum Rates
Ethernet Background Information
To get started, some background information is appropriate. Communication between computer systems using TCP/IP takes place through the exchange of packets. A packet is a PDU (Protocol Data Unit) at the IP layer. The PDU at the TCP layer is called a segment while a PDU at the data-link layer (such as Ethernet) is called a frame. However the term packet is generically used to describe the data unit that is exchanged between TCP/IP layers as well as between two computers.
First, one needs to know the maximum performance of the network environment to establish a baseline value. We will be using a Switched Gigabit Ethernet (IEEE 802.3ab) network configuration. We chose this configuration because it is a typical network topology used in today's enterprise inter-networking environments.
Frames per second (FPS), Packets per second (PPS) or Bits per second (bps) are a common methods of rating the throughput performance of a network device. Understanding how to calculate these rates can provide extensive insight on how the Ethernet system functions and will help assist in network architecture design.
The following sections show the theoretical maximum frames per second that can be achieved using Fast Ethernet (100 Mb/sec) and Gigabit Ethernet (1000 Mb/sec) for the TCP/IP and UDP network protocols. Today's network switches and systems configured with commodity based network NIC adapters allow these theoretical limits to be reached.
Fast Ethernet Using TCP/IP
The diagram below presents maximum Fast Ethernet (IEEE 802.3u) metrics and reference values for the TCP/IP network protocol using minimum and maximum payload sizes.
Fast Ethernet Using UDP
The diagram below presents maximum Fast Ethernet (IEEE 802.3u) metrics and reference values for the UDP network protocol using minimum and maximum payload sizes.
Gigabit Ethernet Using TCP/IP
The diagram below presents maximum Gigabit Ethernet (IEEE 802.3ab) metrics and reference values for the TCP/IP network protocol using minimum and maximum payload sizes.
Gigabit Ethernet Using UDP
The diagram below presents maximum Gigabit Ethernet (IEEE 802.3ab) metrics and reference values for the UDP network protocol using minimum and maximum payload sizes.
Ethernet Frame Rate & Data Measurements Network Configuration
The Gigabit network diagram below is used as the reference network for performance measurements made throughout this article. Each system shown below is a commodity based system that can be purchased at a local retailer. Both the emachine and the shopper2 system are configured as NST probes and attached to the Gigabit Ethernet network via a Dualcomm DGCS-2005L Gigabit Switch TAP. This allows port: 5 of the TAP to be used as a SPAN port for system vortex to perform network packet capture on interface: "p1p1". System configurations, functionality and network performance measurement tools with command line arguments are also displayed. A separate out-of-band management network is configured for NST WUI usage and Shell command line access.
Packet Generators, Monitors & Capture Tools
pktgen
pktgen is a Linux packet generator that can produce network packets at very high speed in the kernel. The tool is implemented as a Linux module. NST has package this tool as an RPM with a front-end script and configuration file for easy setup and usage. The help document for the script is shown:
pktgen-nst.sh <[-h | --help] | [-i | --module-info] | [-k | --kernel-thread-status] | [-n | --nic-status [network interface]] | [[-v | --verbose] -l | --load-pktgen] | [[-v | --verbose] -u | --unload-pktgen] | [[-v | --verbose] -r | --reset-pktgen] | [[-v | --verbose] -c | --conf <pktgen-nst.conf>]>
Maximum Gigabit Ethernet Data Rate Measurement
trafgen: UDP 1514 Byte Packets
trafgen 0.5.6.0 CFG: n 1000000, gap 0 us, pkts 1 [0] pkt len 1514 cnts 0 rnds 0 payload ff ff ff ff ff ff fe 00 00 00 00 00 08 00 45 00 05 ce 12 34 40 00 ff 11 d9 d0 c0 a8 04 64 c0 a8 04 65 07 d0 07 d1 05 ba 87 ec 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 TX: 238.38 MB, 3814 Frames each 65536 Byte allocated MD: FIRE RR 10us Running! Hang up with ^C! 1000000 frames outgoing 1514000000 bytes outgoing [root@emachine tmp]# [root@probe ~]#
netsniff-ng 0.5.6.0 RX: 238.41 MB, 122064 Frames each 2048 Byte allocated OUI UDP TCP ETH PROMISC BPF: (000) ret #-1 MD: RX SCATTER/GATHER B 5 1514 1317192007.975538 B 5 1514 1317192007.975550 B 5 1514 1317192007.975556 . . . B 5 1514 1317192020.280075 B 5 1514 1317192020.280076 B 5 1514 1317192020.280077 1000000 frames incoming 1000000 frames passed filter 0 frames failed filter (out of space) [root@shopper2 shm]#
/dev/shm [root@shopper2 shm]#
total 1497072 drwxrwxrwt 2 root root 60 Sep 26 06:19 . drwxr-xr-x 22 root root 4100 Sep 26 05:55 .. -rw------- 1 root root 1530000024 Sep 28 02:40 c1.pcap [root@shopper2 shm]#
File name: ./c1.pcap File type: Wireshark/tcpdump/... - libpcap File encapsulation: Ethernet Packet size limit: file hdr: 65535 bytes Number of packets: 1000000 File size: 1530000024 bytes Data size: 1514000000 bytes Capture duration: 12 seconds Start time: Wed Sep 28 02:40:07 2011 End time: Wed Sep 28 02:40:20 2011 Data byte rate: 123044025.00 bytes/sec Data bit rate: 984352199.97 bits/sec Average packet size: 1514.00 bytes Average packet rate: 81270.82 packets/sec SHA1: 2107c313d8ac4634fd4552f3809c15c7f4d4550d RIPEMD160: 8358815a4a5db76a1a862a1375bc413a21b4fe16 MD5: e8f6ae140d2905f9b313e114951158b7 Strict time order: True [root@shopper2 shm]#
iperf: TCP/IP 1514 Byte Packets
------------------------------------------------------------ Client connecting to 192.168.4.101, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.4.100 port 42002 connected with 192.168.4.101 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec [root@emachine tmp]#
------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.4.101 port 5001 connected with 192.168.4.100 port 42002 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
Offload parameters for p1p1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off [root@shopper2 shm]#
netsniff-ng 0.5.6.0 RX: 238.41 MB, 122064 Frames each 2048 Byte allocated OUI UDP TCP ETH PROMISC BPF: (000) ret #-1 MD: RX < 5 74 1317200168.297910 > 5 74 1317200168.297932 < 5 66 1317200168.298025 < 5 90 1317200168.298042 > 5 66 1317200168.298050 < 5 4410 1317200168.298202 < 5 5858 1317200168.298255 < 5 4410 1317200168.298299 > 5 66 1317200168.298316 > 5 66 1317200168.298331 > 5 66 1317200168.298348 < 5 1514 1317200168.298432 > 5 66 1317200168.298446 < 5 5858 1317200168.298487 < 5 1514 1317200168.298494 < 5 4410 1317200168.298533 > 5 66 1317200168.298538 > 5 66 1317200168.298548 > 5 66 1317200168.298560 < 5 5858 1317200168.298585 21 frames incoming 21 frames passed filter 0 frames failed filter (out of space) [root@shopper2 shm]#
------------------------------------------------------------ Client connecting to 192.168.4.101, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.4.100 port 42002 connected with 192.168.4.101 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec [root@emachine tmp]#
------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ 5] local 192.168.4.101 port 5001 connected with 192.168.4.100 port 42011 [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
Offload parameters for p1p1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off [root@shopper2 shm]#
netsniff-ng 0.5.6.0 RX: 238.41 MB, 122064 Frames each 2048 Byte allocated OUI UDP TCP ETH PROMISC BPF: (000) ret #-1 MD: RX < 5 74 1317200253.911661 > 5 74 1317200253.911683 < 5 66 1317200253.911780 < 5 90 1317200253.911798 > 5 66 1317200253.911805 < 5 1514 1317200253.911936 < 5 1514 1317200253.911958 < 5 1514 1317200253.911969 < 5 1514 1317200253.911972 < 5 1514 1317200253.911975 < 5 1514 1317200253.911977 < 5 1514 1317200253.911980 > 5 66 1317200253.911995 > 5 66 1317200253.912001 > 5 66 1317200253.912007 > 5 66 1317200253.912011 > 5 66 1317200253.912015 < 5 1514 1317200253.912019 < 5 1514 1317200253.912021 > 5 66 1317200253.912027 49 frames incoming 49 frames passed filter 0 frames failed filter (out of space) [root@shopper2 shm]#
Maximum Gigabit Ethernet Frame Rate Measurement
trafgen: UDP 60 Byte Packets
Ethernet Framing - Segmentation & Checksum (CRC) Offloading
Ethernet Flow Control Pause Frame (IEEE 802.3x)