Date: Fri, 14 Oct 2005 22:59:56 +1000 From: Michael VInce <mv@roq.com> To: freebsd-net@freebsd.org Cc: stable@freebsd.org Subject: Network performance 6.0 with netperf Message-ID: <434FABCC.2060709@roq.com>
next in thread | raw e-mail | index | archive | help
Hey all, I been doing some network benchmarking using netperf and just simple 'fetch' on a new network setup to make sure I am getting the most out of the router and servers, I thought I would post some results in case some one can help me with my problems or if others are just interested to see the results. The network is currently like this, where machines A and B are the Dell 1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM on it) and server B is the gateway and A is acting as a client for fetch and netperf tests. A --- B --- C The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is running 5.4-stable i386 from Oct 12 My main problem is that if I compile SMP into the machine C (5.4stable) the network speed goes down to a range between 6mbytes/sec to 15mbytes/sec on SMP. If I use GENERIC kernel the performance goes up to what I have show below which is around 65megabytes/sec for a 'fetch' get test from Apache server and 933mbits/sec for netperf. Does any know why why network performance would be so bad on SMP? Does any one think that if I upgrade the i386 SMP server to 6.0RC1 the SMP network performance would improve? This server will be running java so I need it to be stable and is the the reason I am using i386 and Java 1.4 I am happy with performance of direct machine to machine (non SMP) which is pretty much full 1gigabit/sec speeds. Going through the gateway server-B seems to drop its speed down a bit for in and out direction tcp speed tests using netperf I get around 266mbits/sec from server A through gateway Server-B to server-C which is quite adequate for the link I currently have for it. Doing a 'fetch' get for a 1gig file from the Apache server gives good speeds of close to 600mbits/sec but netperf shows its weakness with 266mbits/sec. This is as fast as I need it to be but does any one know the weak points on the router gateway to make it faster? Is this the performance I should expect for FreeBSD as a router with gigabit ethers? I have seen 'net.inet.ip.fastforwarding' in some peoples router setups on the list but nothing about what it does or what it can affect. I haven't done any testing with polling yet but if I can get over 900mbits/sec on the interfaces does polling help with passing packets from one interface to the other? All machines have PF running other then that they don't really have any sysctls or special kernel options. Here are some speed benchmarks using netperf and 'fetch' gets. Server A to server C with server C using SMP kernel and just GENERIC kernel further below B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 57344 57344 4096 10.06 155.99 tank# fetch -o - > /dev/null http://server-C/file1gig.iso - 100% of 1055 MB 13 MBps 00m00s ##### Using generic non SMP kernel Server A to server C with server C using GENERIC kernel. A# fetch -o - > /dev/null http://server-C/file1gig.iso - 100% of 1055 MB 59 MBps 00m00s A# ./tcp_stream_script server-C /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 57344 57344 4096 60.43 266.92 ------------------------------------ ############################################### Connecting from server-A to B (gateway) A# ./tcp_stream_script server-B ------------------------------------ /usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 57344 57344 4096 61.80 926.82 ------------------------------------ ########################################## Connecting from server B (gateway) to server C Fetch and Apache2 test B# fetch -o - > /dev/null http://server-C/file1gig.iso - 100% of 1055 MB 74 MBps 00m00s Netperf test B# /usr/local/netperf/tcp_stream_script server-C /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 57344 57344 4096 62.20 933.94 ------------------------------------ Cheers, Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?434FABCC.2060709>