Date: Wed, 19 Oct 2005 23:20:00 +1000 From: Michael VInce <mv@roq.com> To: Robert Watson <rwatson@FreeBSD.org> Cc: freebsd-net@freebsd.org, stable@freebsd.org Subject: Re: Network performance 6.0 with netperf Message-ID: <43564800.3010309@roq.com> In-Reply-To: <20051014205434.C66245@fledge.watson.org> References: <434FABCC.2060709@roq.com> <20051014205434.C66245@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson wrote: > > On Fri, 14 Oct 2005, Michael VInce wrote: > >> I been doing some network benchmarking using netperf and just simple >> 'fetch' on a new network setup to make sure I am getting the most out >> of the router and servers, I thought I would post some results in >> case some one can help me with my problems or if others are just >> interested to see the results. > > > Until recently (or maybe still), netperf was compiled with -DHISTOGRAM > by our port/package, which resulted in a significant performance > drop. I believe that the port maintainer and others have agreed to > change it, but I'm not sure if it's been committed yet, or which > packages have been rebuilt. You may want to manually rebuild it to > make sure -DHISTOGRAM isn't set. > > You may want to try setting net.isr.direct=1 and see what performance > impact that has for you. > > Robert N M Watson I reinstalled the netperf to make sure its the latest. I have also decided to upgrade Server-C (the i386 5.4 box) to 6.0RC1 and noticed it gave a large improvement of network performance with a SMP kernel. As with the network setup ( A --- B --- C ) with server B being the gateway, doing a basic 'fetch' from the gateway (B) to the Apache server (C) it gives up to 700mbits/sec transfer performance, doing a fetch from server A thus going through the gateway gives slower but still decent performance of up to 400mbits/sec. B> fetch -o - > /dev/null http://server-c/file1gig.iso - 100% of 1055 MB 69 MBps 00m00s A> fetch -o - > /dev/null http://server-c/file1gig.iso - 100% of 1055 MB 39 MBps 00m00s Netperf from the gateway directly to the apache server (C) 916mbits/sec B> /usr/local/netperf/netperf -l 20 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 Elapsed Throughput - 10^6bits/sec: 916.50 Netperf from the client machine through the gateway to the apache server (C) 315mbits/sec A> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 57344 Elapsed Throughput - 10^6bits/sec: 315.89 Client to gateway netperf test shows the direct connection between these machines is fast. 912mbits/sec A> /usr/local/netperf/netperf -l 30 -H server-B -t TCP_STREAM -i 10,2 -I 99,5 -- -m 4096 -s 57344 -S 5734 Elapsed Throughput - 10^6bits/sec: 912.11 The strange thing now is in my last post I was able to get faster speeds from server A to C with 'fetch' tests on non-smp kernels and slower speeds with netperf tests. Now I get speeds a bit slower with fetch tests but faster netperf speed tests with or without SMP on server-C. I was going to test with 'net.isr.dispatch' but the sysctl doesn't appear to exist, doing this returns nothing. 'sysctl -a | grep 'net.isr.dispatch' I also tried polling but its also like that doesn't exist either. ifconfig em3 inet 192.168.1.1 netmask 255.255.255.224 polling ifconfig: polling: Invalid argument When doing netperf tests there was high interrupt usage. CPU states: 0.7% user, 0.0% nice, 13.5% system, 70.0% interrupt, 15.7% idle Also the server B is using its last 2 gigabit ethernet ports which are listed from pciconf -lv as '82547EI Gigabit Ethernet Controller' While the first 2 are listed as 'PRO/1000 P' Does any one know if the PRO/1000P would be better? em0@pci5:4:0: class=0x020000 card=0x118a8086 chip=0x108a8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'PRO/1000 P' em3@pci9:8:0: class=0x020000 card=0x016d1028 chip=0x10768086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = '82547EI Gigabit Ethernet Controller' Cheers, Mike > >> >> The network is currently like this, where machines A and B are the >> Dell 1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM >> on it) and server B is the gateway and A is acting as a client for >> fetch and netperf tests. >> A --- B --- C >> The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is >> running 5.4-stable i386 from Oct 12 >> >> My main problem is that if I compile SMP into the machine C >> (5.4stable) the network speed goes down to a range between >> 6mbytes/sec to 15mbytes/sec on SMP. >> If I use GENERIC kernel the performance goes up to what I have show >> below which is around 65megabytes/sec for a 'fetch' get test from >> Apache server and 933mbits/sec for netperf. >> Does any know why why network performance would be so bad on SMP? >> >> Does any one think that if I upgrade the i386 SMP server to 6.0RC1 >> the SMP network performance would improve? This server will be >> running java so I need it to be stable and is the the reason I am >> using i386 and Java 1.4 >> >> I am happy with performance of direct machine to machine (non SMP) >> which is pretty much full 1gigabit/sec speeds. >> Going through the gateway server-B seems to drop its speed down a bit >> for in and out direction tcp speed tests using netperf I get around >> 266mbits/sec from server A through gateway Server-B to server-C which >> is quite adequate for the link I currently have for it. >> >> Doing a 'fetch' get for a 1gig file from the Apache server gives good >> speeds of close to 600mbits/sec but netperf shows its weakness with >> 266mbits/sec. >> This is as fast as I need it to be but does any one know the weak >> points on the router gateway to make it faster? Is this the >> performance I should expect for FreeBSD as a router with gigabit ethers? >> >> I have seen 'net.inet.ip.fastforwarding' in some peoples router >> setups on the list but nothing about what it does or what it can affect. >> I haven't done any testing with polling yet but if I can get over >> 900mbits/sec on the interfaces does polling help with passing packets >> from one interface to the other? >> All machines have PF running other then that they don't really have >> any sysctls or special kernel options. >> >> Here are some speed benchmarks using netperf and 'fetch' gets. >> >> Server A to server C with server C using SMP kernel and just GENERIC >> kernel further below >> >> B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 >> -I 99,5 -- -m 4096 -s 57344 -S 57344 >> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 57344 57344 4096 10.06 155.99 >> tank# fetch -o - > /dev/null http://server-C/file1gig.iso >> - 100% of 1055 MB 13 >> MBps 00m00s >> >> ##### Using generic non SMP kernel >> Server A to server C with server C using GENERIC kernel. >> A# fetch -o - > /dev/null http://server-C/file1gig.iso >> - 100% of 1055 MB 59 >> MBps 00m00s >> >> A# ./tcp_stream_script server-C >> >> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I >> 99,5 -- -m 4096 -s 57344 -S 57344 >> >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 57344 57344 4096 60.43 266.92 >> >> ------------------------------------ >> ############################################### >> Connecting from server-A to B (gateway) >> A# ./tcp_stream_script server-B >> >> ------------------------------------ >> >> /usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I >> 99,5 -- -m 4096 -s 57344 -S 57344 >> >> TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 57344 57344 4096 61.80 926.82 >> >> ------------------------------------ >> ########################################## >> Connecting from server B (gateway) to server C >> Fetch and Apache2 test >> B# fetch -o - > /dev/null http://server-C/file1gig.iso >> - 100% of 1055 MB 74 >> MBps 00m00s >> >> Netperf test >> B# /usr/local/netperf/tcp_stream_script server-C >> >> /usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I >> 99,5 -- -m 4096 -s 57344 -S 57344 >> >> TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 57344 57344 4096 62.20 933.94 >> >> ------------------------------------ >> >> Cheers, >> Mike >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43564800.3010309>