Date: Wed, 3 Feb 2016 17:18:40 +0000 From: "Meyer, Wolfgang" <wolfgang.meyer@hob.de> To: 'Allan Jude' <allanjude@freebsd.org> Cc: "'freebsd-net@FreeBSD.org'" <freebsd-net@FreeBSD.org> Subject: RE: ixgbe: Network performance tuning (#TCP connections) Message-ID: <EC88118611AE564AB0B10C6A4569004D0137D57DA4@HOBEX11.hob.de> In-Reply-To: <56B2216B.6080404@freebsd.org> References: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de> <56B2216B.6080404@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message----- > From: Allan Jude [mailto:allanjude@freebsd.org] > Sent: Mittwoch, 3. Februar 2016 16:49 > To: Meyer, Wolfgang > Subject: Re: ixgbe: Network performance tuning (#TCP connections) > > On 2016-02-03 08:37, Meyer, Wolfgang wrote: > > Hello, > > > > we are evaluating network performance on a DELL-Server (PowerEdge > R930 with 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) > with 10 GbE-Cards. We use programs that on server side accepts connection= s > on a IP-address+port from the client side and after establishing the > connection data is sent in turns between server and client in a predefine= d > pattern (server side sends more data than client side) with sleeps in > between the send phases. The test set-up is chosen in such way that every > client process initiates 500 connections handled in threads and on the se= rver > side each process representing an IP/Port pair also handles 500 connectio= ns > in threads. > > > > The number of connections is then increased and the overall network > througput is observed using nload. On FreeBSD (on server side) roughly at > 50,000 connections errors begin to occur and the overall throughput won't > increase further with more connections. With Linux on the server side it = is > possible to establish more than 120,000 connections and at 50,000 > connections the overall throughput ist double that of FreeBSD with the sa= me > sending pattern. Furthermore system load on FreeBSD is much higher with > 50 % system usage on each core and 80 % interrupt usage on the 8 cores > handling the interrupt queues for the NIC. In comparison Linux has <10 % > system usage, <10 % user usage and about 15 % interrupt usage on the 16 > cores handling the network interrupts for 50,000 connections. > > > > Varying the numbers for the NIC interrupt queues won't change the > performance (rather worsens the situation). Disabling Hyperthreading > (utilising 40 cores) degrades the performance. Increasing MAXCPU to utili= se > all 80 cores won't improve compared to 64 cores, atkbd and uart had to be > disabled to avoid kernel panics with increased MAXCPU (thanks to Andre > Oppermann for investigating this). Initiallly the tests were made on 10.2 > Release, later I switched to 10 Stable (later with ixgbe driver version 3= .1.0) > but that didn't change the numbers. > > > > Some sysctl configurables were modified along the network performance > guidelines found on the net (e.g. > https://calomel.org/freebsd_network_tuning.html, > https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html, > https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of > them didn't have any measuarable impact. Final sysctl.conf and loader.con= f > settings see below. Actually the only tunables that provided any > improvement were identified to be hw.ix.txd, and hw.ix.rxd that were > reduced (!) to the minimum value of 64 and hw.ix.tx_process_limit and > hw.ix.rx_process_limit that were set to -1. > > > > Any ideas what tunables might be changed to get a higher number of TCP > connections (it's not a question of the overall throughput as changing th= e > sending pattern allows me to fully utilise the 10Gb bandwidth)? How can I > determine where the kernel is spending its time that causes the high CPU > load? Any pointers are highly appreciated, I can't believe that there is = such a > blatant difference in network performance compared to Linux. > > > > Regards, > > Wolfgang > > > > <loader.conf>: > > cc_htcp_load=3D"YES" > > hw.ix.txd=3D"64" > > hw.ix.rxd=3D"64" > > hw.ix.tx_process_limit=3D"-1" > > hw.ix.rx_process_limit=3D"-1" > > hw.ix.num_queues=3D"8" > > #hw.ix.enable_aim=3D"0" > > #hw.ix.max_interrupt_rate=3D"31250" > > > > #net.isr.maxthreads=3D"16" > > > > <sysctl.conf>: > > kern.ipc.soacceptqueue=3D1024 > > > > kern.ipc.maxsockbuf=3D16777216 > > net.inet.tcp.sendbuf_max=3D16777216 > > net.inet.tcp.recvbuf_max=3D16777216 > > > > net.inet.tcp.tso=3D0 > > net.inet.tcp.mssdflt=3D1460 > > net.inet.tcp.minmss=3D1300 > > > > net.inet.tcp.nolocaltimewait=3D1 > > net.inet.tcp.syncache.rexmtlimit=3D0 > > > > #net.inet.tcp.syncookies=3D0 > > net.inet.tcp.drop_synfin=3D1 > > net.inet.tcp.fast_finwait2_recycle=3D1 > > > > net.inet.tcp.icmp_may_rst=3D0 > > net.inet.tcp.msl=3D5000 > > net.inet.tcp.path_mtu_discovery=3D0 > > net.inet.tcp.blackhole=3D1 > > net.inet.udp.blackhole=3D1 > > > > net.inet.tcp.cc.algorithm=3Dhtcp > > net.inet.tcp.cc.htcp.adaptive_backoff=3D1 > > net.inet.tcp.cc.htcp.rtt_scaling=3D1 > > > > net.inet.ip.forwarding=3D1 > > net.inet.ip.fastforwarding=3D1 > > net.inet.ip.rtexpire=3D1 > > net.inet.ip.rtminexpire=3D1 > > > > > > > > > > ________________________________ > > > > Follow HOB: > > > > - HOB: http://www.hob.de/redirect/hob.html > > - Xing: http://www.hob.de/redirect/xing.html > > - LinkedIn: http://www.hob.de/redirect/linkedin.html > > - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html > > - Facebook: http://www.hob.de/redirect/facebook.html > > - Twitter: http://www.hob.de/redirect/twitter.html > > - YouTube: http://www.hob.de/redirect/youtube.html > > - E-Mail: http://www.hob.de/redirect/mail.html > > > > > > HOB GmbH & Co. KG > > Schwadermuehlstr. 3 > > D-90556 Cadolzburg > > > > Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic > > > > AG Fuerth, HRA 5180 > > Steuer-Nr. 218/163/00107 > > USt-ID-Nr. DE 132747002 > > > > Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org" > > > > Is there a reason you are disabling TSO? I would expect that to have a > significant impact on throughput. It is usually only disabled on a router= , to > reduce latency on forwarded packets. > > net.inet.tcp.tso=3D1 > > Also, please provide the list with the output of 'ifconfig ix0' > > -- > Allan Jude This sysctl was one of the knobs I found in the internet that should be con= sidered to improve network performance. Actually it won't change the number= s either way (just tested again). ifconfig ix1: ix1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=3D8407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VL= AN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO> ether a0:36:9f:7c:9e:52 inet 172.31.1.1 netmask 0xffffff00 broadcast 172.31.1.255 inet 172.31.2.1 netmask 0xffffff00 broadcast 172.31.2.255 inet 172.31.3.1 netmask 0xffffff00 broadcast 172.31.3.255 inet 172.31.4.1 netmask 0xffffff00 broadcast 172.31.4.255 nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active Regards, Wolfgang Meyer ________________________________ Follow HOB: - HOB: http://www.hob.de/redirect/hob.html - Xing: http://www.hob.de/redirect/xing.html - LinkedIn: http://www.hob.de/redirect/linkedin.html - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html - Facebook: http://www.hob.de/redirect/facebook.html - Twitter: http://www.hob.de/redirect/twitter.html - YouTube: http://www.hob.de/redirect/youtube.html - E-Mail: http://www.hob.de/redirect/mail.html HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EC88118611AE564AB0B10C6A4569004D0137D57DA4>