The internet is getting faster and faster. I now have Gigabit Fiber delivered to within a foot of my home router. The carrier then is kind enough to provide a smart jack that converts the fiber delivery to a Gigabit Copper handoff.
More bandwidth is better right? Well kinda… Unfortunately, I hadn’t planned on one thing I am now much more vulnerable to denial of service attacks. Why is this an issue? Whats an engineer to do? Read on..
The bottleneck in the router
Routers historically are really just purpose built Processors for processing packets with some RAM to buffer (temporarily store) the contents of these packets, PHY and MAC controllers (ignoring technologies outside of ethernet) for receipt and delivery of packets, and a bus for interconnecting all of these internals. Note that the PHY and MAC are usually combined into a single modular card represented as the Ethernet NIC (Network Interface Card) in a commodity PC/Server.
Even Cisco, Juniper, Brocade, or any other major manufacturer’s routing hardware, short of being highly proprietary in both expansion and parts, are fairly trivial from a high level implementationwise. Switching is a completely different story but I digress. The point is, forwarding performance is largely based on the slowest portion of the router, the Processor.
So if you look at the flow of packet through a router, in simple generic terms, voltage on the wire are recognized by the PHY using line coding int a serial bit string. The MAC (Media Access Controller) recognizes the Ethernet preamble and begins storing the bytes following the preamble in a register (small on controller memory location). When the Packet completes successful reception, the packet is copied into main memory (RAM) over the bus (functionally similar to PCI/PCI-Express) and notifies the processor via an interrupt that a packet has arrived. The processor then executes a small set of prepared instructions (a callback) which examines the packet, determines what to do with it, and then tells the MAC to write the bytes containing the packet in main memory to the PHY which again encodes the bitstream into signals using the appropriate line coding scheme.
We as humans talk about the capacity of our internet connection as bit rate (commonly, bits of transfer per second), but in actuality the router is limited by its processor in how fast it can process a packet regardless of the packet’s size. So basically a 64 byte (the smallest legitimate IP Packet), 1500 byte packet or a 9216 byte (jumbo frame) packet puts the same amount of stress on the router. All this to say, a router’s true maximum throughput is measured in packet rate… and the size of the packets traversing the router really are what determine whether the router can meet the expectation of the human’s utilizing said router.
Just to show how important it is to know the limits of your router in packets per second lets look at some calculations of throughput based on various packet sizes for my home router specifically. I own a Cisco 1941W. It can forward 299,000 packets per second according to Cisco’s published “Router Performance Matrix.”
Upper limits for different sized packets:
Large MTU packets: 1500 byte packets
299,000 packets / 1 second
* 1,500 bytes / 1 packet
* 8 bits / 1 byte
---------------------------------------------------------------
3,588,000,000 bits / 1 second (~3.5 Gbps max throughput)
IMIX MTU packets: (Approximation based on real world packets)
40 bytes * 7 packets
+ 576 bytes * 4 packets
+ 1,500 bytes * 1 packet
---------------------------------
4,084 bytes / 12 packets
299,000 packets / 1 second
* 4,084 bytes / 12 packets
* 8 bits / 1 byte
---------------------------------------------------------------
813,280,000 bits / 1 second (~800 Mbps max throughput)
Small MTU packets: 64 byte packets
299,000 packets / 1 second
* 64 bytes / 1 packet
* 8 bits / 1 byte
---------------------------------------------------------------
19,136,000 bits / 1 second (~20 Mbps max throughput)
And just to show how misleading the numbers can get, imagine if all the vendors did their estimation of router throughput based on Jumbo frame sizes…
Jumbo MTU packets: 9216 byte packets
299,000 packets / 1 second
* 9,216 bytes / 1 packet
* 8 bits / 1 byte
---------------------------------------------------------------
22,044,672,000 bits / 1 second (~22 Gbps max throughput)
So theoretically, my little 1941 could do 22 Gbps through it.
Higher rate interface risk and reward
So, just in case it isn’t readily obvious: my router should be able to handle IMIX traffic of nearly 800 mbps. If all traffic and endpoints on the internet were good and righteous I wouldn’t be concerned. The reality is it only takes one nefarious endpoint and around 19 Mbps of traffic to overwhelm my router and prevent this wonderful gigabit connection from slowing to a trickle.
It is always a good idea when purchasing routers, (especially on behalf of your employer) to make sure that the router is capable of processing more packets per second than your interfaces can receive. This prevents the nethers of the ether from clogging up your router’s ability to what it does best… forward packets. So in this instance, I only have one connection to the internet and that connection can deserialize data off of the wire at a rate of 1,073,741,824 bits per second. (Note that if you have multiple carriers attached to the same router, you should sum the interface bandwidth of all of those carriers.) Back to our trusty calculation.. lets figure out how much packet processing we need:
1,073,741,824 bits / 1 second
* 1 packet / 64 bytes
* 1 byte / 8 bits
---------------------------------------------------------------
2,097,152 packets / 1 second (minimum)
So, with a router capable of 2 Mpps I should be at the very least capable of keeping my router from becoming the choke point. Hopefully my upstream carrier will notice an absurd amount of 64 byte packets and resolve the issue prior to me having to take action.
What router to buy?
So, great.. lets find a router. Google tell me, what is the ‘fastest home office router.’ First up, PC Magazine’s ‘The Best Wireless Routers of 2018‘, second up c|net’s ‘Best Wireless Routers for 2018 and last good ole faithful, Tom’s Hardware with ‘Best Wifi Routers 2018</>’
Note: It is fairly difficult to find information with the pps ratings for soho routers, and in fact there was no review site I could find that mentioned anything but throughput at what is presumed to be IMIX in Mbps. So I kept my research simple: Start with the higher end first, which coincides with also being relatively expensive for the common home user. But I wanted pure unadulterated throughput, so lets see whats out there.
Asus RT-AC88U @ $229 USD
- Branded Gigabit Router
- Broadcom BCM7409C0 1.4 GHz dual-core processor (ARM Cortex A9)
- Boasts 1800 Mbps of throughput on their website
- 360,000 pps
- IMIX packets @ 980 Mbps
- Small packets @ 184 Mbps
Linksys WRT1900ACS @ 169 USD
- Marvell Armada 385 (MV88F6820) 1.6 GHz dual-core processor (ARM Cortex A9)
This router in every test I could find outperformed the Linksys EA9200 (Linksys’ $299 router), and looks like it maxes out at around 650 Mbps on a wired connection. I didn’t bother purchasing it to test it as the theoretical maximum packet forwarding capacity is 300 kpps according to multiple sources.
Netgear Nighthawk X10 R9000 AD7200 @ $429 USD
- Annapurna Labs Alpine AL-514 quad-core 1.7 GHz Processor
- 7.2 Gbps Advertised throughput
- 1,400,000 Mpps
- IMIX packets @ 3.8 Gbps
- Small packets @ 716 Mbps
So it appears there isn’t really a home router capable of mitigating DoS attacks when the carrier is providing gigabit access to the internet.. at least as far as I could find. So I did some research in an arena I’m a little more familiar with and started looking at used enterprise hardware.
Cisco uBR7200-NPE-G2 @ $479 USD (adding $71.99 for Chassis) via Cables and Kits
- Only 1,300,000 Mpps
Cisco ASR 1001-X @ $7,145 USD via Cables and Kits
- 14 Mpps
- IMIX packets @ 38 Gbps
- Small packets @ 7 Gbps
So it looks like there is a serious lack of equipment that can manage the packet load of small packets on a gigabit internet interface without costing a fortune. About this time I was dabbling in just using a higher end PC (Dell Precision 5500) as a router. The advantage being, I’m very familiar with FreeBSD and its network stack and am quite capable of manually setting it up as long as the hardware could move packets fast enough while performing the firewall function.
Custom FreeBSD Router @ $750 USD via ebay
- Dual quad-core Intel CPU at 2.4 GHz
- 16 GB of RAM
- Chelsio TS540-CR NIC @ $345 USD via Amazon
So I backed up my workstation and built a fresh FreeBSD 12 HEAD install on it. Walked through this fine tutorial and kicked off some performance tests to see where I was at… Turns out this works quite well, short of this being the physically largest gigabit capable router I’ve seen. I could consistently get 3 Mpps with ipfw enabled. But honestly, I just wasn’t willing to give up my workstation to be the household router. There had to be something else… however in the process I had found many many things that excited me about what I had just done. Before this is all over I will have my next router run FreeBSD. But I digress.. off to find a more streamlined option.. hopefully something that will fit in the closet.
Ubiquiti EdgeRouter Pro 8
In reading on the forums for FreeBSD for my BSD Router attempt, I found several setups interesting, I nearly purchased a PC Engines APU2 Board as it seemed to be more than capable of accomplishing what I needed, before coming across a FreeBSD install on the EdgeRouter Lite. It performed well considering the Edge Router’s small form factor and relatively weak processor, and thats when it hit me. It uses an Octeon MIPS64 proc, which I believe has a builtin packet acceleration processor.
But the EdgeRouter Lite was too weak at 1 Mpps I was about halfway where I wanted to be, so I started digging into what other options Ubiquiti had and sure enough they have an EdgeRouter 8 and EdgeRouter Pro 8 which both handle 2 Mpps. The best part, it clocked in pricewise less than the Nighthawk at $360 on Amazon.
So I purchased it. I will say that EdgeOS in and of itself is pretty impressive. I would not say this product is a fast replacement featurewise for a Cisco router. However, the GUI is slick and the webpage provides access to the CLI without having to use SSH or a console cable. It has met all my goals. I can successfully move without loss 940 Mbps of 64 byte packets through the router and the CPU only hits about 80% utilization, which is quite a bit of headroom, and best yet the CLI and GUI is completely responsive under load.
So of course the next thing I had to do was attempt to boot FreeBSD on it.. and unfortunately the ERP8 utilizes the Octeon II CPU whereas the ERL used the Octeon+ CPU, resulting in the ERP8 not being capable of booting an ERL image. So now I have yet another project to take on. Feel free to keep apprised of my work going forward on extending the FreeBSD MIPS support onto the Octeon II in the hopes to very shortly be able to boot FreeBSD on the ERP8 and have the ultimate SDN box.
protocolSyntax 