Something interesting I learned (and I still believe it to be true, but I wouldnt be surprised if modern network tech can negate it):
Minimum packet length ~= Maximum network segment
This is to prevent two distant stations from transmitting on the same network segment, writing the complete packet and not listening for a collision, and then having the two packets collide in the middle of the segment.
(Min packet length) x (Network speed e.g. 100Mbps) x (line transmission speed) = Maximum network length
20% is so much that the gain can’t be blamed on the packet overhead. The difference in packet overhead between jumbos and regular frames isn’t even 1%. I bet that the difference is in the TCP window back-off.
Looking at the two graphs, I see that the max throughput is about the same for both. The regular frames, however, backoff farther down that the jumbo frames. That is what is killing performance, IMHO.
If you have a way to monitor TCP window size over time I bet that you’ll see the difference in the graphs. If the connection is relatively free of noise you could also increase the retransmit time-out to improve performance. You could run Wireshark and snoop out a few TCP packets to get a feel for the TCP window size over time.
@JMJ AFAIK you’re right on classic/old ethernet network: hub and/or coaxial calble (10Mbit). If you use switches (and not hubs) the collision domain is reduced to two point: you and the switch. The switch uses internal buffer and every port is a collision domain.
Jumbo Frames made a HUGE difference for throughput on my home network. I went from 30’ish MB/sec to roughly 50MB/sec. The only downside is I game a lot and this can cause more latency. I enabled it on my other vista-64 box and my home server and they xfer much better now (I’m not using JF on my gaming rig). Great read!
The most interesting thing about this is the very low usage of the network capacity.
60MB/s*8bits = 480Mbits/s
Not counting the SMB (Windows networking) header, the ethernet/IP/UDP only accounts for an additional 3% (or less for jumbo frames)
The problem is Window’s SMB protocol. You can get typically get DOUBLE the speed (i.e. a 100% improvement) using FTP, which is much better than the 26% improvement Jeff achieved using Jumbo frames.
Jumbo frames are great. I work on VMware ESX networking, and I will point out what may not be obvious to everyone. In a virtualized environment (hosted or hypervisor) jumbo frames make an even bigger difference, since you are doing more work per packet to begin with. That’s why we added jumbo frame support since ESX 3.5 shipped.
My experience is that any recent machine can easily push full 1Gbit line rate (on native, and for that matter ESX VMs). Setting Jumbo Frames will save you on CPU though, which will allow you to run more VMs or otherwise use that power. And while Jumbo Frames are nice- they get you from 1.5k packets to 9, TCP Segmentation Offloading (TSO) is much better, since you push down entire 64k (or sometimes up to 256k) packets, and an engine on the NIC itself automatically handles dicing them into 1.5K packets. Most good nics support this- Intel, Broadcom, etc. On the other side, the reverse is LRO, or RSS, but this is more complicated and less common. Plus with TSO, you don’t have to worry about MTU.
The other thing I would mention is- for the love of god, don’t run networking benchmarks by doing a file copy. With 1GBit networks, you are limited by your disk speed! Run a proper tool such as iperf (brain dead simple) or netperf, which just blasts data. Even if your hard drive could reach 1Gbit speeds, you would be wasting cycles, so your networking performance would be worse. You always want to look at these things in isolation.
Sorry… as I read more misinformed posts, I feel the need to lay down more truth.
The reason that all these people are seeing performance improvements using Jumbo Frame on Windows is because Windows networking stack sucks. Windows is really stupid and often will not let a single tcp stream reach the full capacity of the NIC. I.e. you run 1 TCP stream and measure 400Mbits, but if you ran 3 in parallel you would hit 940Mbits (~Line rate). This is even more annoying with 10G, since you need like 18 streams to reach the peak performance. Linux doesn’t have these problems, and will give you its best possible performance on a single stream. I can only imagine Window’s behavior is the result of some misguided attempt at ensuring fairness between connections by making sure that even if there is only one connection, it never uses the full capacity.
If you simply enable jumbo frames on your NIC, every connection to any Internet destination (which don’t support jumbos) will need to undergo PMTU discovery, PMTU blackhole detection, router fragmentation, or other time-consuming / performance-sapping hacks. This might explain why people complain about latency issues with gaming. These people are also seeing slightly slower peformance with all Internet activity.
*nix, as/400/, mainframes, and other operating systems let you set the frame size on a per route basis. E.g.,
route add -net 0.0.0.0 gw 192.168.0.1 mss 1460
This tells the OS to use jumbo frames only on the local LAN, and to assume a normal packet size everywhere else.
Alas, Windows has no such ability. One solution on Windows is to use two NICs attached to the same network. Have one NIC configured with normal frames and the default route. Have the second NIC configured for jumbos with no default route.
Pardon the off-topic comment, but in a sense, it’s in place. I think that although this is a technical discussion, I’d prefer working harder at programming and becoming a technical expert about data, interaction, speeds and things like that, rather than blogging about the social aspects or hacking, although I am quite successful for my tiny brain size (hurray!) . Those things are far more fluid than data packets. So in that sense experimenting with Jumbo size packets is not for me. There! Back on topic.
Now there’s another big decision - monkey business or the light of human freedom? Or both? Or the popular second-grade coffee?
I participated in the IEEE 802.3 committee for a while. IEEE never standardized a larger frame size for two reasons that I know of:
The end stations can negotiate the frame size, but there was no backwards-compatible way to ensure that all L2 bridges between them can handle it. Even if you send a jumbo frame successfully, you can still run into a problem later if the network topology changes and your packets begin taking a different path through the network.
The CRC32 at the end of the packet becomes weaker after around 4 KBytes of data. It can no longer guarantee that single bit errors will be caught, and the multibit error detection becomes weaker as well.
One is free to enable it, and it does improve the performance, but the situation is unlikely to ever get better in terms of standard interoperability. It will always be an option to be enabled manually.
Also a number of years ago,. jumbo frames provided a much bigger boost. Going from 1.5K to 9K regularly doubled performance or more. What has happened since is smarter ethernet NICs: they routinely coalesce interrupts, steer packets from the same flow to the same CPU, and sometimes even reassemble the payload of the 1.5K frames back into larger units. The resistance to standardizing jumbo frames resulted in increased innovation elsewhere to compensate.
transmitting 9KB of data, or 72 kilobit, would take 72 microseconds. That’s not problematic latency, and a 1000 times as much still wouldn’t be (for voice, anyhow).
Intuitively, if 1.5KB was OK on 10MB, then the same latency should be achievable with 150KB frames on a gigabit.
Of course we’d also like to capitalize on the increased switching speed so going that far is overkill - but frame-size related latency isn’t something we need to worry about - of course, low latency apps shouldn’t buffer up to a full frame, so there might be a software issue here.
@Timothy Layer 2 ethernet switches will just drop packets they cannot handle. It is not just if they don’t handle jumbo frames: they can drop a normal size packet if their internal queues are full, or if rate limiting has been configured, or if the switch hit some other internal condition which the ASIC designer didn’t bother resolving. They just drop the packet and expect the sender to retransmit. There is no mechanism for an L2 device to send back a notification that it has dropped the packet. A managed L2 switch will have some counters so you can post-mortem analyze what is wrong with your network.
Layer 3 routers will drop packets for more reasons, in addition to queue congestion. For example when the packet is too big and the don’t fragment bit is set, an ICMP message is sent back (this is how path MTU discovery works). Similarly routers send back ICMP messages if they have no route to the destination.
Even the ICMP is just a best effort notification. Routers routinely limit the rate of ICMP messages they will send, to avoid having a flood of ICMP messages make a bad network problem worse. ICMP messages can also be dropped on their way back to the originator. So the best the sender can expect is it might get notified of an L3 problem, sometimes.
Let’s not forget about Ethernet Pause frames, which can help to avoid frames being dropped by switches that have filled up their port buffers. A previous comment suggested there was no layer 2 message to indicate drops, but Pause frames aka xon/off are a layer 2 protocol.
As a point of interest, in 2015 I see about 117 MB/sec of network throughput on gigabit (with jumbo frames enabled, naturally) using a modern router and modern PCs.
That’s almost 2x the 60 MB/sec I measured in 2009!
Humm. I was reading this post to enable Jumbo Frames for my network.
I cannot upload an image here (got an error saying new users cannot upload images). But I constantly get 110 MB/s to 120 MB/s between my desktop and server not doing anything - all using Windows version of networking drivers, and going through 2 Gigabit switches. Everything set to factory MTU 1500.
I haven’t set anything related to Jumbo Frames.
Asus RIVBE mobo - 1 Gbps connection w/Windows 8.1 (Windows drivers for network)
1x Netgear 8 port 1 Gbps switch
1x Linksys 5 port 1 Gbps switch
MSI X58 mobo - 1 Gbps connection w/Windows Server 2012 (Windows default drivers)
(about 150 feet of wiring between them all)
I think Jumbo frames are more for 10 Gbps connections?
. Those things are far more fluid than data packets. So in that sense experimenting with Jumbo size packets is not for me. There! Back on topic. 
