At work today, we had a problem with a particular workstation. Although it was connected to a gigabit ethernet hub, network file transfers were "too slow". How do you quantify "too slow"?
I finally unpacked my server, which has a gigabit 64-bit PCI card (it’s a dual Athlon MP system). I’m using the d-link gigabit router described above, and the integrated nForce 4 gigabit on my desktop. The tray icons for both connections show 1.0 Gbps.
I get anywhere from ** 33-35 megabytes/sec ** when running a TTCP test from my desktop to the server. That’s in the ballpark…
–
D:\Storage\ttcppcattcp -t -f M homeserver
PCAUSA Test TCP Utility V2.01.01.08
TCP Transmit Test
Transmit : TCP - 192.168.0.10:5001
Buffer Size : 8192; Alignment: 16384/0
TCP_NODELAY : DISABLED (0)
Connect : Connected to 192.168.0.10:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCP - 192.168.0.10:5001
16777216 bytes in 0.48 real seconds = 33.06 MB/sec +++
numCalls: 2048; msec/call: 0.24; calls/sec: 4231.40
Experimenting with larger packet sizes (the default is 8192 bytes):
pcattcp -t -l 16384 -f M homeserver
51.20 MB/sec
pcattcp -t -l 32768 -f M homeserver
69.41 MB/sec
pcattcp -t -l 65536 -f M homeserver
71.23 MB/sec
pcattcp -t -l 131072 -f M homeserver
69.72 MB/sec
Gigabit ethernet tops out at ~70 megabytes per second, at least for pure memory read/writes.
The old 64-bit PCI gig-E card went kaput. The new card is a single chip design and it does substantially better in the same server on the same exact network-- nothing else has changed:
pcattcp -t -f M homeserver
34.12 MB/sec
pcattcp -t -l 16384 -f M homeserver
64.00 MB/sec
pcattcp -t -l 32768 -f M homeserver
93.16 MB/sec
pcattcp -t -l 65536 -f M homeserver
105.00 MB/sec
pcattcp -t -l 131072 -f M homeserver
103.73 MB/sec
Er… wow. About the same with small 8kb packets, but WAYYY better with larger ones! Which is a good thing, because all this donkey porn isn’t going to copy itself from the server. That’s all I’m saying…
According to the Windows documentation, the TCP/IP default window size is 17,520 bytes. That’s a reasonable setting for gig-e, but not optimal.
http://support.microsoft.com/kb/q224829/
It auto-negotiates up to 65kb. This can be improved, though, per this article which looks quite consistent with the kb doc above.
http://www.enterpriseitplanet.com/networking/features/article.php/3497796
Here’s how to enable large frames in the TCP/IP stack in Windows XP, Windows 2000, and Windows Server 2003:
Edit the registry via regedit.exe and navigate to
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Increase the TCP/IP window size to 132k and enable rfc1323 scaling and timestamps:
increase the TCP buffer sizes:
I put these four tweaks in a standalone registry file for your convenience:
http://www.codinghorror.com/blog/files/gigabit_large_window.zip
NumForwardPackets should NOT be set to more than ForwardBufferMemory / 256!
http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/en-us/regentry/58793.asp
Also, do a google on Numforwardpackets.
So, set ForwardBufferMemory to 79872, divide by 256 and that would give NumForwardPackets a max of 312? Does that sound about right?
NumForwardPackets should NOT be set to more than ForwardBufferMemory / 256!
if my calculations are correct, 80000 /256 60000 !!
Is this a typo ?
I looked in http://www.codinghorror.com/blog/files/gigabit_large_window.zip and it’s not a typo. I’d like to add that ForwardBufferMemory must be a multiple of 256 and the default value is 74,240.
Remember that if the program is copying the data to HDD, there will your bottleneck be. The system will probably not transfer faster than it needs to, so your HDD write speed will be the determining factor. (I think).
Well your hard drive may seem to be bottlenecking as it is the slowest component of the computer, but many modern desktop hard drives can read/write on average 50 MB/s (that’s Big M Big B for megabytes a second). Personally, my 500 gig sata2 samsung 7200rpm can read/write from 60 to 80 MB/s bursting up to 100 MB/s or more. Even my older IDE 7200 rpm can transfer at least 30 MB/s. Speaking of bottlenecks, USB hard drives are being bottlenecked to 25-30 MB/s as a result of USB HI-speed’s low sustained bandwidth(theoretical burst speeds up to 480 Mb/s but sustains around 300 Mb/s). This is one reason why I’d prefer a file server with gigabit connections rather than a JBOD configuration of USB hard drives.
Jan 2008 now…
Just tested with OS X 10.5.1 between a PowerMac G5 and a PowerBook G4 both connected via Gigabit (standard mtu of 1500) to a NETGEAR GS105. I’m getting 70MB/sec which is much faster than the rule of thumb posted above (this is between two machines not in memory on a single machine).
ttcp -r -s -f M
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.190
ttcp-r: 1677721600 bytes in 22.36 real seconds = 71.57 MB/sec +++
ttcp-r: 234073 I/O calls, msec/call = 0.10, calls/sec = 10470.32
ttcp-r: 0.2user 5.0sys 0:22real 23% 0i+0d 0maxrss 0+0pf 223322+0csw
–
My in memory tests report 455 MB/sec and 128 MB/sec.
Computer 1 (PowerMac G5)
ttcp -r -s -f M
ttcp -t -s -n 710400 -f M localhost
ttcp-t: buflen=8192, nbuf=710400, align=16384/0, port=5001 tcp - localhost
ttcp-t: socket
ttcp-t: connect
ttcp-t: 5819596800 bytes in 12.18 real seconds = 455.55 MB/sec +++
ttcp-t: 710400 I/O calls, msec/call = 0.02, calls/sec = 58310.97
ttcp-t: 0.5user 11.6sys 0:12real 99% 0i+0d 0maxrss 0+0pf 15+0csw
Computer 2 (PowerBook G4)
ttcp -r -s -f M
ttcp -t -s -n 305200 -f M localhost
ttcp-t: buflen=8192, nbuf=305200, align=16384/0, port=5001 tcp - localhost
ttcp-t: socket
ttcp-t: connect
ttcp-t: 2500198400 bytes in 18.57 real seconds = 128.39 MB/sec +++
ttcp-t: 305200 I/O calls, msec/call = 0.06, calls/sec = 16433.41
ttcp-t: 0.5user 5.5sys 0:18real 32% 0i+0d 0maxrss 0+0pf 813+0csw
Great article, very useful. Am actually looking at an industrial application that requires about 50MB/s - 80 MB/s data rates and I’m investigating whether gigabit ethernet could be a viable (cheap) solution.
However I have one query, when you refer to changing the packet size, I presume you don’t actual mean the packet size but rather the amount of data transmitted? If this is the case the speed difference is probably to due to efficiencies of scale.
I thought the packet size (MTU) defaulted to 1500, or 9kB (jumbo frames).
I just tried benchmarking my home network. All my pc is connected to a gigabit ethernet router. Gigabit ethernet is really a lot fast.
Using pcattcp -t -l 131072 -f M MyPC, I can get about 108-109 MB/s which is about 864-872 Mbps. For file transfer from PC1-PC2, i can get about 38MB/s.
The PCATTCP download has a virus in it.
…picked up by Avast.
I just wrote 70MB/s across gigabit, so I’m incredibly happy.
Yep. I have achieved over 100MB/s sustained writing from RAM to RAM over gigabit ethernet, and this includes any SAMBA overhead.
I have never used pcattcp, but I do use iperf and it confirms that I am getting the full speeds once should expect from gigabit ethernet, even on cheap shitty Realtek chips.
My guess is that something else fails in this test. Maybe old wires not good for gigabit, or a shitty switch or something.
I ran the test again, I had to use two console windows:
In the first window
pcattcp.exe -r
In the second window
pcattcp -t -f M localhost
results
TCP Transmit Test
Transmit : TCPv4 0.0.0.0 -> 127.0.0.1:5001
Buffer Size : 8192; Alignment: 16384/0
TCP_NODELAY : DISABLED (0)
Connect : Connected to 127.0.0.1:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCPv4 0.0.0.0 -> 127.0.0.1:5001
16777216 bytes in 0.033 real seconds = 483.56 MB/sec +++
numCalls: 2048; msec/call: 0.017; calls/sec: 61895.551
So, ten years later, we go from 90 MB/sec to 484 MB/sec in loopback ethernet testing. Real world throughput from machine to machine these days in 2015 is about 120 MB/sec max.