Meidokon Rants

Science and anime and languages and stuff

Understanding actual switching capacity of network switches

I was recently comparing the performance of Mikrotik and Arista switches by looking at their spec sheets. This is not a fair or meaningful comparison by any stretch, but it's what I've got - I have lots of Mikrotik kit at home, and I work on Arista switches five days a week for my job.

Any switch worth its salt can forward packets at full line rate, full duplex on every port, in optimal conditions. But what are optimal conditions, and what does "line rate" actually mean?

Let's assume a switch with 8x 1Gbps ports. "Optimal conditions" usually means that packets are flowing in one port and out another, there are no corrupted or malformed packets, and that the packets are large enough. With standard 1500 byte packets, you only need to send 83,333 packets per second to reach 1Gbps (we'll call this 83.3 kpps).

If you shrink the packets to 500 bytes each, the switch now needs to handle 250,000 packets per second (250 kpps) to reach 1Gbps of throughput, which is a lot more work. As packet sizes decrease the switch will have to do a lot more processing and forwarding work just to maintain the same throughput, and eventually it'll run out of steam. This is why spec sheets usually quote kpps figures for a few different packet sizes.

For this investigation I'm looking at the:

  • Arista 7010T-48, a modest 48-port 1Gbps top-of-rack switch [spec sheet]
  • Mikrotik CRS320-8P-8B-4S+RM, a 16-port 1Gbps PoE switch aimed at Wifi APs etc [product page]

The key thing here is that these are both switches, not routers, and we care about their packet-forwarding performance. The difference in port-count doesn't matter. Incidentally, both switches have 4x 10G SFP ports for uplinking to your network core.

Throughput calculations and full duplex

Before we get too deep into the weeds we need to lay out some assumed knowledge.

  • A gigabit port can push 1Gbps in each direction, and full-duplex means it can do both directions at the same time - that's 2Gbps of "capacity" according to Mikrotik, or "throughput" according to Arista.
  • A 10-gigabit port can obviously do 10Gbps each way, making for 20Gbps of capacity in a full-duplex scenario.
  • The Arista 7010 has 48x 1G ports (48Gbps of throughput) and 4x 10G ports (40Gbps throughput), for a total of 88Gbps throughput, or 176Gbps of full-duplex capacity, which matches Arista's performance claim.
  • The Mikrotik CRS320 actually has 17x 1G ports (17Gbps of throughput, port 17 is an extra mgmt port) and 4x 10G ports (40Gbps throughput), for a total of 57Gbps throughput, or 114Gbps of full-duplex capacity. This is what we see on the last two lines of the published table.

The Mikrotik switch

Mikrotik is nice, they publish a table of performance test figures:

According to the table we can get 57G of Layer 1 throughput at all packet sizes. Let's just sanity-check that...

57G == 57,000,000,000 bits == 7,125,000,000 bytes
7125000000 / 64 bytes packets = 111,328,125 packets
They say 84,821.4kpps, which is  84,821,400 packets, so there's a discrepancy

# the top line makes sense though
84821400 packets * 64 bytes = 5,428,569,600 bytes == 43,428,556,800 bits

That last number matches the top-right cell, being 43,428 Mbps.

Let's just check the Layer 2 numbers with 1518-byte packets:

1518 bytes == 12144 bits
It's rated for 4,632.6kpps == 4,632,600 pps
4,632,600 packets * 12144 bits per packet = 56,258,294,400 bits per second

That's really close to 57G of throughput! In fact it's exactly their quoted L2 throughput once you take rounding and truncation into account: 56,258.8 Mbps

The Layer 1 number (57,000 Mbps) is 1.013174828x the Layer 2 number (56,258.8 Mbps), which suggests an extra 20 bytes per frame, ie. 1538 bytes, inflated from 1518 bytes. Wait, that sounds like the size of an IPv4 header. Or could it be something else..?

To understand this we need to look at the structure of ethernet frames and how they're put onto the wire: https://en.wikipedia.org/wiki/Ethernet_frame#Structure

Lo and behold, the Layer 1 framing actually adds another 20 bytes! It's 8 bytes for the preamble at the start of the frame, and 12 bytes for the interpacket gap. The preamble is so the receiver can get a "lock" on the sender's clock signal, and the IPG is to give receivers time to prepare for receiving the next frame (it's not technically data, it's just a wait that's 12 bytes in length if it were transmitted).

Let's check the other values in the table for Layer 2 performance.

512-byte frame == 4096-bit frame
4096 bits * 13,392,846 pps = 54,857.1 Mbps - this is correct

64-byte frame == 512-bit frame
43,428,600,000 bps / 512 bits = 84,821,484 pps - this looks correct tooCode language: JavaScript (javascript)

So in an optimal situation, your 1518-byte Layer 2 frame costs 1538 "bytes" to put onto the wire, which is an overhead of 1.318%. In a worst-ish case scenario, your 64-byte Layer 2 frame costs 84 bytes to put on the wire, which is a horrendous overhead; your Layer 2 payload is only 76% of the Layer 1 structure.

Incidentally, the Mbps throughput for 64-byte packets is about 76% of that for 1518-byte packets, so I think that's where it's all going. It's not that the switch can't keep up with small packets, it's that they're simply that inefficient that you lose lots of performance.

This is, I think, the real explanation as to why switches lose throughput performance at smaller packet sizes. At least to some extent - if it drops to less than 76% throughput with 64-byte packets then it's a packet-rate issue, not a small-packet-efficiency issue.

Back to that performance table for a moment, I think we can reason about the weird numbers it shows.

It looks like the top line is fine (L2 throughput and capacity), as is the 1518 byte column, it's just the L1 rows that are wrong for non-1518-byte frames. They should get smaller as the packet size decreases, due to overheads of transmission. Actually the packet rates are fine, it's just the Mbps figures that are wrong in the lower-right section of the table.

The Arista switch

Arista claims:

  • 176Gbps of throughput
  • 132 million packets per second

The 176Gbps of throughput (or "capacity" in Mikrotik terms) will assume 1500-byte packets, while the 132 million pps figure is likely quoted for 64-byte packets. It's trivial to calculate that:

132,000,000 packets * 64 bytes * 8 bits/byte = 67,584,000,000 bps, or 67.584Gbps

Actually I'm somewhat confused by the Capacity pps numbers in Mikrotik's tables - shouldn't they be 2x the Throughput numbers? You can't have twice the bandwidth and not be pushing twice as many packets.

Anyway we now know that those 64-byte packets actually occupy 84 bytes in terms of wire performance. Let's redo that calculation:

132,000,000 packets * 84 bytes * 8 bits/byte = 88,704,000,000 bps, or 88.704Gbps

With 48x 1G ports and 4x 10G ports, I think this is actually overstated. We should expect to see 88Gbps of L1 switching performance, and this is about 0.7% too high.

If we start with 88Gbps and work backwards, we arrive at 130,952,380.952 pps, or 130.95 mpps.

Back To Top