From owner-freebsd-performance@freebsd.org Wed Feb 3 13:38:43 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69546A993DF; Wed, 3 Feb 2016 13:38:43 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from hobex19.hob.de (hobex29.hob.de [212.185.199.32]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D086B13DD; Wed, 3 Feb 2016 13:38:38 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from HOBEX22.hob.de (172.22.1.4) by hobex29.hob.de (172.25.1.32) with Microsoft SMTP Server (TLS) id 14.2.347.0; Wed, 3 Feb 2016 14:37:10 +0100 Received: from HOBEX11.hob.de ([fe80::b99f:1c54:7122:49b4]) by HOBEX22.hob.de ([::1]) with mapi id 14.02.0387.000; Wed, 3 Feb 2016 14:37:22 +0100 From: "Meyer, Wolfgang" To: "'freebsd-net@FreeBSD.org'" CC: "'freebsd-performance@FreeBSD.org'" Subject: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67w== Date: Wed, 3 Feb 2016 13:37:21 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.24.71.140] old-x-esetresult: clean, is OK old-x-esetid: 46BD7B39450636371DFD21 x-esetresult: clean, is OK x-esetid: 46BD7B39450636371DFD21 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2016 13:38:43 -0000 Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://calomel.org/freebsd_network_tuning.htm= l, https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html, ht= tps://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of them did= n't have any measuarable impact. Final sysctl.conf and loader.conf settings= see below. Actually the only tunables that provided any improvement were i= dentified to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minim= um value of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that w= ere set to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang : cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" : kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: http://www.hob.de/redirect/hob.html - Xing: http://www.hob.de/redirect/xing.html - LinkedIn: http://www.hob.de/redirect/linkedin.html - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html - Facebook: http://www.hob.de/redirect/facebook.html - Twitter: http://www.hob.de/redirect/twitter.html - YouTube: http://www.hob.de/redirect/youtube.html - E-Mail: http://www.hob.de/redirect/mail.html HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 From owner-freebsd-performance@freebsd.org Wed Feb 3 17:50:31 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0F88A975AB for ; Wed, 3 Feb 2016 17:50:31 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 941F51DC7 for ; Wed, 3 Feb 2016 17:50:31 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 84824D05D for ; Wed, 3 Feb 2016 17:50:30 +0000 (UTC) Subject: Re: ixgbe: Network performance tuning (#TCP connections) To: freebsd-performance@freebsd.org References: From: Allan Jude Message-ID: <56B23DEA.1060307@freebsd.org> Date: Wed, 3 Feb 2016 12:50:34 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LhlsgOkJpnPBQLLK0bCCsMJ9mmIm32k3i" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2016 17:50:31 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --LhlsgOkJpnPBQLLK0bCCsMJ9mmIm32k3i Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-02-03 08:37, Meyer, Wolfgang wrote: > Hello, >=20 > we are evaluating network performance on a DELL-Server (PowerEdge R930 = with 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with= 10 GbE-Cards. We use programs that on server side accepts connections on= a IP-address+port from the client side and after establishing the connec= tion data is sent in turns between server and client in a predefined patt= ern (server side sends more data than client side) with sleeps in between= the send phases. The test set-up is chosen in such way that every client= process initiates 500 connections handled in threads and on the server s= ide each process representing an IP/Port pair also handles 500 connection= s in threads. >=20 > The number of connections is then increased and the overall network thr= ougput is observed using nload. On FreeBSD (on server side) roughly at 50= ,000 connections errors begin to occur and the overall throughput won't i= ncrease further with more connections. With Linux on the server side it i= s possible to establish more than 120,000 connections and at 50,000 conne= ctions the overall throughput ist double that of FreeBSD with the same se= nding pattern. Furthermore system load on FreeBSD is much higher with 50 = % system usage on each core and 80 % interrupt usage on the 8 cores handl= ing the interrupt queues for the NIC. In comparison Linux has <10 % syste= m usage, <10 % user usage and about 15 % interrupt usage on the 16 cores = handling the network interrupts for 50,000 connections. >=20 > Varying the numbers for the NIC interrupt queues won't change the perfo= rmance (rather worsens the situation). Disabling Hyperthreading (utilisin= g 40 cores) degrades the performance. Increasing MAXCPU to utilise all 80= cores won't improve compared to 64 cores, atkbd and uart had to be disab= led to avoid kernel panics with increased MAXCPU (thanks to Andre Opperma= nn for investigating this). Initiallly the tests were made on 10.2 Releas= e, later I switched to 10 Stable (later with ixgbe driver version 3.1.0) = but that didn't change the numbers. >=20 > Some sysctl configurables were modified along the network performance g= uidelines found on the net (e.g. https://calomel.org/freebsd_network_tuni= ng.html, https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.= html, https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most o= f them didn't have any measuarable impact. Final sysctl.conf and loader.c= onf settings see below. Actually the only tunables that provided any impr= ovement were identified to be hw.ix.txd, and hw.ix.rxd that were reduced = (!) to the minimum value of 64 and hw.ix.tx_process_limit and hw.ix.rx_pr= ocess_limit that were set to -1. >=20 > Any ideas what tunables might be changed to get a higher number of TCP = connections (it's not a question of the overall throughput as changing th= e sending pattern allows me to fully utilise the 10Gb bandwidth)? How can= I determine where the kernel is spending its time that causes the high C= PU load? Any pointers are highly appreciated, I can't believe that there = is such a blatant difference in network performance compared to Linux. >=20 > Regards, > Wolfgang >=20 I wonder if this might be NUMA related. Specifically, it might help to make sure that the 8 CPU cores that the NIC queues are pinned to, are on the same CPU that is backing the PCI-E slot that the NIC is in. --=20 Allan Jude --LhlsgOkJpnPBQLLK0bCCsMJ9mmIm32k3i Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJWsj3tAAoJEBmVNT4SmAt+CgIP/2pcubjW1rIy5T91UFIuztD7 kNj5hF7H5C8/h3U998OwCMPAgRJKfnOjDybVaHb3QNwGYKsMiAUietVbFTF4k928 oR8HxD/xuNYMpdicVeB48winvEOA8ilaFjbaKYE4dNX7XaNk/5+610PrRRkwJfuE dN96ycI83nrEMJFHJFFu88bnfrjX0oP26AlqsN2G8hqExQwPVERgLniNN59cx/9x 4aUQO8w4W62PWvc63x5/rjMVKBjlnnAxt89kGq6doCseLWkrdnQHhr2rlha3vEG5 zEQyPTcoN+kb2wxzpnSBWppxZ9mGdCDmz2HyLMD0nF8ypTgJIMl4OuhfPPWHS+i4 ogRQkq3UWyUtjQhd+ZF8lXGYkzyDR1yf9vAS7r/RxkJFCcvr7SVZO+ADDzkMbt9S jz+re8irH+pV1fAEA4NFYPeQL8ngs96Y37sjH/nZovuUDT6elUmwBcMxzbQTC44r c0xpPfkcH8c0lqg6OYUrIqoQULVjCr5swriq6lb8ygfpLHzcgyAeZHiAyQGo0fDb ieXI9uJ2RGXVYslX+cCfxypz22Y7ULfKzQjeeuf4Z80DdX7AnfvgOqeRikncFv6Q zg+dNc2dbSOs5zywPYScmSpFZjVMh1MJG7g1Y8Uig1BY5+JJ7Bqztis9i9drCmyE YDu7R9zyqEr2VAKVy2s/ =nUC0 -----END PGP SIGNATURE----- --LhlsgOkJpnPBQLLK0bCCsMJ9mmIm32k3i-- From owner-freebsd-performance@freebsd.org Wed Feb 3 19:30:39 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05DDEA9AA63 for ; Wed, 3 Feb 2016 19:30:39 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C1DE01A1C; Wed, 3 Feb 2016 19:30:38 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-io0-x235.google.com with SMTP id d63so67742539ioj.2; Wed, 03 Feb 2016 11:30:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=vp6fLgzIMSx/Ij/wTj/cRU9vhfxp2qGpoYbbyhAyk2s=; b=PbJheKKiGiJGPioIiHTO6MioKzdzT4DdxJFWWc1vmYTN879b6njk5SIqRitQci0NHS P0ZFJNadjz+/w8SlL8xSoywsx3JaX2RzUs7EJXu7Ji8xwH7HvqvCsD9hT7ZkXVOHhTP6 gwHpo68yQMPkmkWYSXBft24v9jd+oJcnRuK9XpL/fPmTtvzfwMqyctDWYRB5BU/rtBM5 AOdVzGjnFxs/ADYovV5Q0V7oo9CH6FiXD2i4h61LUA8OXUiWtTTo0asOySDFzgO/EKaQ lrDdi1VBvqCJyhT11tKhBibdwzcLSzVGBpJJSx+/mxXDtEa5uPoknkdj7VXhSKf1a4tP RUAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=vp6fLgzIMSx/Ij/wTj/cRU9vhfxp2qGpoYbbyhAyk2s=; b=k7YoGgPAzNcHCWyAEUXI0IvLZerN6DjOqdCs2P+ohGmJCK/Fjt1vhSqtjHBDF8P+YJ n3U+0kJJO775HmiJ5LBw32/7PYmuRmBm94AeqT3FMtAOeo+6zHkw2LkcqxwjzZq/i2AT WNDepHmweZx8DbWwsZypuCAS1y2oFGh/YrQrtMy65pNZWb2qcEcnsqtspxmR5w/b7GKN iBEspL9MHWGlzW1GuA0GvmZd13xlkktv/rEasaa1UF5ovafoM7xa49AEk5ctTMN/XVkh rhB+K1Ond9/QgtvherQfLOeZ6uwmYx2zm4U5uFYLZpzwB95Kr1lqdRoohRnOPB4gDYFJ G+BA== X-Gm-Message-State: AG10YOQkhYZr1fFvRzeg+pVdupYb70BwP05DYa1vyVSUGAAqWOOovV4N7nWQCivHhLi0uYkW9qKth7HqF8MTCg== MIME-Version: 1.0 X-Received: by 10.107.38.82 with SMTP id m79mr5102377iom.2.1454527837993; Wed, 03 Feb 2016 11:30:37 -0800 (PST) Sender: kmacybsd@gmail.com Received: by 10.36.15.10 with HTTP; Wed, 3 Feb 2016 11:30:37 -0800 (PST) In-Reply-To: <56B23DEA.1060307@freebsd.org> References: <56B23DEA.1060307@freebsd.org> Date: Wed, 3 Feb 2016 11:30:37 -0800 X-Google-Sender-Auth: aKQQZJ1JMrCHFKqlceDe2DU2KDA Message-ID: Subject: Re: ixgbe: Network performance tuning (#TCP connections) From: "K. Macy" To: Allan Jude Cc: "freebsd-performance@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2016 19:30:39 -0000 Also - check for txq overruns and rxq drops in sysctl. 64 is very low on FreeBSD. You may also look in to increasing the size of your pcb hash table. On Wed, Feb 3, 2016 at 9:50 AM, Allan Jude wrote: > On 2016-02-03 08:37, Meyer, Wolfgang wrote: >> Hello, >> >> we are evaluating network performance on a DELL-Server (PowerEdge R930 w= ith 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10= GbE-Cards. We use programs that on server side accepts connections on a IP= -address+port from the client side and after establishing the connection da= ta is sent in turns between server and client in a predefined pattern (serv= er side sends more data than client side) with sleeps in between the send p= hases. The test set-up is chosen in such way that every client process init= iates 500 connections handled in threads and on the server side each proces= s representing an IP/Port pair also handles 500 connections in threads. >> >> The number of connections is then increased and the overall network thro= ugput is observed using nload. On FreeBSD (on server side) roughly at 50,00= 0 connections errors begin to occur and the overall throughput won't increa= se further with more connections. With Linux on the server side it is possi= ble to establish more than 120,000 connections and at 50,000 connections th= e overall throughput ist double that of FreeBSD with the same sending patte= rn. Furthermore system load on FreeBSD is much higher with 50 % system usag= e on each core and 80 % interrupt usage on the 8 cores handling the interru= pt queues for the NIC. In comparison Linux has <10 % system usage, <10 % us= er usage and about 15 % interrupt usage on the 16 cores handling the networ= k interrupts for 50,000 connections. >> >> Varying the numbers for the NIC interrupt queues won't change the perfor= mance (rather worsens the situation). Disabling Hyperthreading (utilising 4= 0 cores) degrades the performance. Increasing MAXCPU to utilise all 80 core= s won't improve compared to 64 cores, atkbd and uart had to be disabled to = avoid kernel panics with increased MAXCPU (thanks to Andre Oppermann for in= vestigating this). Initiallly the tests were made on 10.2 Release, later I = switched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn= 't change the numbers. >> >> Some sysctl configurables were modified along the network performance gu= idelines found on the net (e.g. https://calomel.org/freebsd_network_tuning.= html, https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html,= https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of them = didn't have any measuarable impact. Final sysctl.conf and loader.conf setti= ngs see below. Actually the only tunables that provided any improvement wer= e identified to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the mi= nimum value of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit tha= t were set to -1. >> >> Any ideas what tunables might be changed to get a higher number of TCP c= onnections (it's not a question of the overall throughput as changing the s= ending pattern allows me to fully utilise the 10Gb bandwidth)? How can I de= termine where the kernel is spending its time that causes the high CPU load= ? Any pointers are highly appreciated, I can't believe that there is such a= blatant difference in network performance compared to Linux. >> >> Regards, >> Wolfgang >> > > I wonder if this might be NUMA related. Specifically, it might help to > make sure that the 8 CPU cores that the NIC queues are pinned to, are on > the same CPU that is backing the PCI-E slot that the NIC is in. > > > -- > Allan Jude > From owner-freebsd-performance@freebsd.org Wed Feb 3 21:34:05 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AE12A99579; Wed, 3 Feb 2016 21:34:05 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9A83D25; Wed, 3 Feb 2016 21:34:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x229.google.com with SMTP id rs20so13533920igc.0; Wed, 03 Feb 2016 13:34:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=iIS/1D5ntVZgOmhLD0yJjosaDNv7pIm8Sp8h4rmub7I=; b=0yX5rpoUOujET51YvqUQ7qj7GMFadteSaLa3mP5mNYbMtLkRU45IvOzqy5ZyDszJ6f 2tRRSVgZHQuofdLW+pSVkb7tiBOLGI6zZQAPg9CnLuiXHIuQq6u99BtD5n0QEyqs+c4E TbX6hRwafILEN9JhHoZtIw6tHMl6KM+pGBiUrS402t+FRg1x42YnZrQU0iPgprxGTVUi iAdkqifSjHN76FD5lMBduK6TJzClc7U069aSjPAp6kIkPb+3zqwMjZcv80MFfXolMkb1 /veHb5mcJieH2222VRedc2uV6c6SjFd2Ytm9ErsxJJO0rItnLg1lnRALvUkoI9Rbqr+z A71w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=iIS/1D5ntVZgOmhLD0yJjosaDNv7pIm8Sp8h4rmub7I=; b=U6wQm/DRGfW1S8CS1UY/ar00wH8lOWEImY5vaWA/bLS57uxipfxB20j23vSZYXCNnv +PrQSBzK+lkmAyjn3al4Ft3kV6eSIpGuaRd+cqz8ED9bbWtLtuAMw5y+TlXlJyY7Oeq7 Xj2omjhc0P5fpzkGpukaTb4Y1ZDeSRQA2SFzonPfq6bOMFRv8xgScQs4az57L2RckP7Z tdcKp6LfML/6rBSBodQuPQF6JVAnCaAk3rO5L53sqNMZVmetR+HmUMimvDc6ig6oY/GO ME1O04FBdNtuEXBx1ZpIV2VWoioSZoNnf7cqN1V8nPN7QoKxyIapebuKie+TL3W8Htle rV5Q== X-Gm-Message-State: AG10YOTgbHq9yimE33TkUWDypNSuPtp8RJlSzuc3n1wUIm7e57Qqb0fPuxjnzYkh7605Yn1TKwZ9WhfmKwOKLQ== MIME-Version: 1.0 X-Received: by 10.50.93.36 with SMTP id cr4mr5566578igb.22.1454535243969; Wed, 03 Feb 2016 13:34:03 -0800 (PST) Received: by 10.36.14.19 with HTTP; Wed, 3 Feb 2016 13:34:03 -0800 (PST) In-Reply-To: References: Date: Wed, 3 Feb 2016 13:34:03 -0800 Message-ID: Subject: Re: ixgbe: Network performance tuning (#TCP connections) From: Adrian Chadd To: "Meyer, Wolfgang" Cc: "freebsd-net@FreeBSD.org" , "freebsd-performance@FreeBSD.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Wed, 03 Feb 2016 21:43:12 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2016 21:34:05 -0000 hi, can you share your testing program source? -a On 3 February 2016 at 05:37, Meyer, Wolfgang wrote: > Hello, > > we are evaluating network performance on a DELL-Server (PowerEdge R930 wi= th 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 = GbE-Cards. We use programs that on server side accepts connections on a IP-= address+port from the client side and after establishing the connection dat= a is sent in turns between server and client in a predefined pattern (serve= r side sends more data than client side) with sleeps in between the send ph= ases. The test set-up is chosen in such way that every client process initi= ates 500 connections handled in threads and on the server side each process= representing an IP/Port pair also handles 500 connections in threads. > > The number of connections is then increased and the overall network throu= gput is observed using nload. On FreeBSD (on server side) roughly at 50,000= connections errors begin to occur and the overall throughput won't increas= e further with more connections. With Linux on the server side it is possib= le to establish more than 120,000 connections and at 50,000 connections the= overall throughput ist double that of FreeBSD with the same sending patter= n. Furthermore system load on FreeBSD is much higher with 50 % system usage= on each core and 80 % interrupt usage on the 8 cores handling the interrup= t queues for the NIC. In comparison Linux has <10 % system usage, <10 % use= r usage and about 15 % interrupt usage on the 16 cores handling the network= interrupts for 50,000 connections. > > Varying the numbers for the NIC interrupt queues won't change the perform= ance (rather worsens the situation). Disabling Hyperthreading (utilising 40= cores) degrades the performance. Increasing MAXCPU to utilise all 80 cores= won't improve compared to 64 cores, atkbd and uart had to be disabled to a= void kernel panics with increased MAXCPU (thanks to Andre Oppermann for inv= estigating this). Initiallly the tests were made on 10.2 Release, later I s= witched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn'= t change the numbers. > > Some sysctl configurables were modified along the network performance gui= delines found on the net (e.g. https://calomel.org/freebsd_network_tuning.h= tml, https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html, = https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of them d= idn't have any measuarable impact. Final sysctl.conf and loader.conf settin= gs see below. Actually the only tunables that provided any improvement were= identified to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the min= imum value of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that= were set to -1. > > Any ideas what tunables might be changed to get a higher number of TCP co= nnections (it's not a question of the overall throughput as changing the se= nding pattern allows me to fully utilise the 10Gb bandwidth)? How can I det= ermine where the kernel is spending its time that causes the high CPU load?= Any pointers are highly appreciated, I can't believe that there is such a = blatant difference in network performance compared to Linux. > > Regards, > Wolfgang > > : > cc_htcp_load=3D"YES" > hw.ix.txd=3D"64" > hw.ix.rxd=3D"64" > hw.ix.tx_process_limit=3D"-1" > hw.ix.rx_process_limit=3D"-1" > hw.ix.num_queues=3D"8" > #hw.ix.enable_aim=3D"0" > #hw.ix.max_interrupt_rate=3D"31250" > > #net.isr.maxthreads=3D"16" > > : > kern.ipc.soacceptqueue=3D1024 > > kern.ipc.maxsockbuf=3D16777216 > net.inet.tcp.sendbuf_max=3D16777216 > net.inet.tcp.recvbuf_max=3D16777216 > > net.inet.tcp.tso=3D0 > net.inet.tcp.mssdflt=3D1460 > net.inet.tcp.minmss=3D1300 > > net.inet.tcp.nolocaltimewait=3D1 > net.inet.tcp.syncache.rexmtlimit=3D0 > > #net.inet.tcp.syncookies=3D0 > net.inet.tcp.drop_synfin=3D1 > net.inet.tcp.fast_finwait2_recycle=3D1 > > net.inet.tcp.icmp_may_rst=3D0 > net.inet.tcp.msl=3D5000 > net.inet.tcp.path_mtu_discovery=3D0 > net.inet.tcp.blackhole=3D1 > net.inet.udp.blackhole=3D1 > > net.inet.tcp.cc.algorithm=3Dhtcp > net.inet.tcp.cc.htcp.adaptive_backoff=3D1 > net.inet.tcp.cc.htcp.rtt_scaling=3D1 > > net.inet.ip.forwarding=3D1 > net.inet.ip.fastforwarding=3D1 > net.inet.ip.rtexpire=3D1 > net.inet.ip.rtminexpire=3D1 > > > > > ________________________________ > > Follow HOB: > > - HOB: http://www.hob.de/redirect/hob.html > - Xing: http://www.hob.de/redirect/xing.html > - LinkedIn: http://www.hob.de/redirect/linkedin.html > - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html > - Facebook: http://www.hob.de/redirect/facebook.html > - Twitter: http://www.hob.de/redirect/twitter.html > - YouTube: http://www.hob.de/redirect/youtube.html > - E-Mail: http://www.hob.de/redirect/mail.html > > > HOB GmbH & Co. KG > Schwadermuehlstr. 3 > D-90556 Cadolzburg > > Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic > > AG Fuerth, HRA 5180 > Steuer-Nr. 218/163/00107 > USt-ID-Nr. DE 132747002 > > Komplementaerin HOB electronic Beteiligungs GmbH > AG Fuerth, HRB 3416 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Wed Feb 3 21:49:28 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DD4BA99E89 for ; Wed, 3 Feb 2016 21:49:28 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0CF174E for ; Wed, 3 Feb 2016 21:49:28 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 8DC14D6DB for ; Wed, 3 Feb 2016 21:49:27 +0000 (UTC) Subject: Re: ixgbe: Network performance tuning (#TCP connections) To: freebsd-performance@freebsd.org References: From: Allan Jude Message-ID: <56B275EC.2060604@freebsd.org> Date: Wed, 3 Feb 2016 16:49:32 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ek6DEQ5gI4ReUpEMS2VCbQRSHu1GiuJoL" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Feb 2016 21:49:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ek6DEQ5gI4ReUpEMS2VCbQRSHu1GiuJoL Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-02-03 16:34, Adrian Chadd wrote: > hi, >=20 > can you share your testing program source? >=20 >=20 > -a >=20 I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel X540-AT2 that I can try to replicate this one to help debug it. --=20 Allan Jude --ek6DEQ5gI4ReUpEMS2VCbQRSHu1GiuJoL Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJWsnXvAAoJEBmVNT4SmAt+sEEP/iZTC9kdibpzPD3sPDYy1wfw QX2SlQPJsns3HrGGAX/Q68uNd2b8MFEbmktQ+rxrzTD6OjFUW1HFeJpZz4G+wHu3 mRRo0iPM9RnvkRw8yRPBMVLwsSYUk/E5A66XWNVWvo8v8wDKnYubO403l1Kw0XKV npw4WcgeN5tYhLtQw3iwvZHbZ0VRJmhmlva5WIBqsisodIB1OvVOISEyJYdjI8GC TlsRvpVW78kUrZ0Nk4U27e9+nESTSLl8XlqTMqH3J4Aju/EL1Cdcg4BYDI5Xw4Db P7LFNratzvdppIXye1nPRg0AYS9YpfWBIrEZ0/v+p7dUZt8jcsvzlR8vStcFdMpn t1Akr2N8V0xohKrMBR2hDru+5MW2yttZSEt98OpjjbuTGTiNuKE0n54ozGj6nM4x F122rzqYxXnSRh+hsuxxIPiUUq7qV1BGmVGXdSfuqy78brqxY5WFggIO8pc2DG5T BXW9CvAxLZUSZw4ek48dWDMRa/ur393oMmP/G/xdzoZk4nV8FvLdZ95kU2P3UWfc LjnAz9bNavzpEEgRYtobCzz1rEkM9YNq1rEXInC4Rfw4illCfdhSq6Qu3HfVcH7J 88qcZ0RzKOW1VznUwLCpMakN2v5OjVHikCEOWQ0pGsYAkQ8Q5H/XLb5xt7I6fzB3 4gNnA/tpjU1zLi5RkOS7 =jH28 -----END PGP SIGNATURE----- --ek6DEQ5gI4ReUpEMS2VCbQRSHu1GiuJoL-- From owner-freebsd-performance@freebsd.org Thu Feb 4 09:09:18 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9A4BA9BDFB; Thu, 4 Feb 2016 09:09:18 +0000 (UTC) (envelope-from remy.nonnenmacher@boostedge.com) Received: from fr-exchange.activnetworks.com (fr-exchange.activnetworks.com [62.210.235.130]) by mx1.freebsd.org (Postfix) with ESMTP id 67C9EA51; Thu, 4 Feb 2016 09:09:17 +0000 (UTC) (envelope-from remy.nonnenmacher@boostedge.com) Received: from rn.activnetworks.com ([192.168.1.100] RDNS failed) by fr-exchange.activnetworks.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 4 Feb 2016 09:39:39 +0100 Subject: Re: ixgbe: Network performance tuning (#TCP connections) To: "Meyer, Wolfgang" , "'freebsd-net@FreeBSD.org'" References: Cc: "'freebsd-performance@FreeBSD.org'" From: Remy Nonnenmacher Message-ID: <56B314F7.3060502@activnetworks.com> Date: Thu, 4 Feb 2016 10:08:07 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 04 Feb 2016 08:39:39.0571 (UTC) FILETIME=[9604B830:01D15F27] X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 09:09:18 -0000 On 02/03/16 14:37, Meyer, Wolfgang wrote: > Hello, > > we are evaluating network performance on a DELL-Server (PowerEdge R930 with 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 GbE-Cards. We use programs that on server side accepts connections on a IP-address+port from the client side and after establishing the connection data is sent in turns between server and client in a predefined pattern (server side sends more data than client side) with sleeps in between the send phases. The test set-up is chosen in such way that every client process initiates 500 connections handled in threads and on the server side each process representing an IP/Port pair also handles 500 connections in threads. > > The number of connections is then increased and the overall network througput is observed using nload. On FreeBSD (on server side) roughly at 50,000 connections errors begin to occur and the overall throughput won't increase further with more connections. With Linux on the server side it is possible to establish more than 120,000 connections and at 50,000 connections the overall throughput ist double that of FreeBSD with the same sending pattern. Furthermore system load on FreeBSD is much higher with 50 % system usage on each core and 80 % interrupt usage on the 8 cores handling the interrupt queues for the NIC. In comparison Linux has <10 % system usage, <10 % user usage and about 15 % interrupt usage on the 16 cores handling the network interrupts for 50,000 connections. > > Varying the numbers for the NIC interrupt queues won't change the performance (rather worsens the situation). Disabling Hyperthreading (utilising 40 cores) degrades the performance. Increasing MAXCPU to utilise all 80 cores won't improve compared to 64 cores, atkbd and uart had to be disabled to avoid kernel panics with increased MAXCPU (thanks to Andre Oppermann for investigating this). Initiallly the tests were made on 10.2 Release, later I switched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't change the numbers. > > Some sysctl configurables were modified along the network performance guidelines found on the net (e.g. https://calomel.org/freebsd_network_tuning.html, https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html, https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of them didn't have any measuarable impact. Final sysctl.conf and loader.conf settings see below. Actually the only tunables that provided any improvement were identified to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set to -1. > > Any ideas what tunables might be changed to get a higher number of TCP connections (it's not a question of the overall throughput as changing the sending pattern allows me to fully utilise the 10Gb bandwidth)? How can I determine where the kernel is spending its time that causes the high CPU load? Any pointers are highly appreciated, I can't believe that there is such a blatant difference in network performance compared to Linux. > > Regards, > Wolfgang > [SNIP] Hi Wolfgang, hwpmc is your friend here if you need to investigate where are your processors wasting their time. Either you will find them contending for network stack (probably the pcb hash table), either they are fighting each other in the scheduler's lock(s) trying to steal jobs from working ones. Also check QPI links activity that may reveal interesting facts about PCI root-complexes geography vs processes locations and migration. You have two options here: Either you persist in using a 4x10 core machine and you will have a long time rearranging stickyness of processes and interrupt to specific cores/packages (Driver, then isr rings, then userland) and police the whole thing (read peacekeeping the riot), either you go to the much simpler solution that is 1 (yes, one) socket machine, fastest available proc with low core (E5-1630v2/3 or 1650) that can handle 10G links hands down out-of-the-box. Also note that there are specific and interesting optimization in the L2 generation on -head that you may want to try if the problem is stack-centered. You may also have a threading problem (userland ones). In the domain of counting instructions per packets (you can practice that with netmap as a wonderfull mean of really 'sensing' what 40Gbps is), threading is bad (and Hyperthreading is evil). Thanks. From owner-freebsd-performance@freebsd.org Thu Feb 4 13:50:17 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59D7FA9C6A2; Thu, 4 Feb 2016 13:50:17 +0000 (UTC) (envelope-from honzhan@microsoft.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0138.outbound.protection.outlook.com [157.56.110.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A53E51D6; Thu, 4 Feb 2016 13:50:15 +0000 (UTC) (envelope-from honzhan@microsoft.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=KUyW6GgtJ2QiBWtpqKpCa2vK5VTlhJsDOdgaXOcbTM4=; b=lInxb4OiHPqEe41gHOhhHGTeiNQgrvzY+0oCHcix1wDreEXmHber3gL/8mfb/arp4qQNQbLGB7SD8YhmjiqfMq8cp1umHYUHsZUmv8CyMb2My2ZG3QAvgWSaFZL1h+c8sQa176o0wyvT3y4zsBf/ntdrvUr9iQRF5X68CA6fWKQ= Received: from BY2PR03CA007.namprd03.prod.outlook.com (10.255.93.24) by BLUPR0301MB1540.namprd03.prod.outlook.com (10.162.213.158) with Microsoft SMTP Server (TLS) id 15.1.396.15; Thu, 4 Feb 2016 13:35:08 +0000 Received: from BL2FFO11FD029.protection.gbl (10.255.93.4) by BY2PR03CA007.outlook.office365.com (10.255.93.24) with Microsoft SMTP Server (TLS) id 15.1.403.16 via Frontend Transport; Thu, 4 Feb 2016 13:35:07 +0000 Authentication-Results: spf=pass (sender IP is 206.191.228.180) smtp.mailfrom=microsoft.com; hob.de; dkim=none (message not signed) header.d=none;hob.de; dmarc=pass action=none header.from=microsoft.com; Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 206.191.228.180 as permitted sender) receiver=protection.outlook.com; client-ip=206.191.228.180; helo=064-smtp-out.microsoft.com; Received: from 064-smtp-out.microsoft.com (206.191.228.180) by BL2FFO11FD029.mail.protection.outlook.com (10.173.160.69) with Microsoft SMTP Server (TLS) id 15.1.409.7 via Frontend Transport; Thu, 4 Feb 2016 13:35:05 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net (141.251.56.18) by SG2PR3002MB0107.064d.mgd.msft.net (141.251.56.19) with Microsoft SMTP Server (TLS) id 15.1.403.10; Thu, 4 Feb 2016 13:35:02 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) by SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) with mapi id 15.01.0403.016; Thu, 4 Feb 2016 13:35:02 +0000 From: Hongjiang Zhang To: "Meyer, Wolfgang" , "'freebsd-net@FreeBSD.org'" CC: "'freebsd-performance@FreeBSD.org'" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAG7Oyw Date: Thu, 4 Feb 2016 13:35:02 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [180.170.80.203] X-MS-Office365-Filtering-Correlation-Id: f5ac5343-39fd-4d76-0e69-08d32d67fe55 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BL2FFO11FD029; 1:vUwuqiTdPhCBH/XtzJYIA9Lcwlqcx4J+2VfiOOZbCWa16t//G/SVZY1ge4wvI+b/7quVVogkTm8KTaS747XU/pBPb0umas9S4ItG0wBr5CEfmBYQsZA43ZKdYJLGe8jIVknyO1uS4GIs0vqaKFhLp8tH/+Vx6Jkrk0pSGV3Yj8NBjC5uAMHoX24864ovKA4lsKATLMf4cmjLEc5Ghu1z1n01X+C1RAhoWlEflcGMAkWmj9Y+4pLkxvjwuNUs2gpu12j2mi9At2W0OXJnfIBBVuN1TjA5Htt8MT3SFPZJesugRqC6Ni/hGMpYSytAK+6yCFywU4d+GbnxWNl/rbxWqA== X-Forefront-Antispam-Report: CIP:206.191.228.180; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(438002)(13464003)(199003)(3905003)(189002)(377454003)(16796002)(50466002)(102836003)(23726003)(1220700001)(1096002)(586003)(3846002)(6116002)(24736003)(10290500002)(10400500002)(189998001)(5001960100002)(5005710100001)(5001770100001)(86612001)(106466001)(97756001)(6806005)(5008740100001)(92566002)(19580395003)(46406003)(108616004)(19580405001)(86146001)(11100500001)(47776003)(87936001)(54356999)(575784001)(86362001)(76176999)(2950100001)(33646002)(66066001)(2900100001)(2906002)(4326007)(5003600100002)(15975445007)(50986999)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:BLUPR0301MB1540; H:064-smtp-out.microsoft.com; FPR:; SPF:Pass; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 2:fTAqvAk/I2b1QK63qoglNfRDa+RFRPq1+cOHwzc74/AqmuNyAypB3oLDIanl/sxQX/0sTg1iiFXsmCLtsr2WcyTcZ8B4gRg9ShZOZURo5feR4QLCt4YDGwj0XW4XP9RmxOtq6qc2IdT9BddoyaErR6QMBUUgkAGnJ0W/+Jj0OaZODhNFeuAd3Db65+DGoCG3; 3:ttkFRxccojWw+0YPG/S6P4geHLpqOpTxjczPntTt5qw1SfZQLU35dSkIcXiSuPAl+gdM/emDwZDXS5dBRTkqZgz1tUyj97HBBogqOxb4CksHa9yIWjGw5tVcXyWbKJNVQIjyuRyPYoQp/g5IqY3TPlI9ybs7Wmyx5ByYBC75ZbK/RrccxI7NuRsFDqDwH6f5IVeuiHpXuVoQ7Yxk/VFCNN0ggCCKA4etuq1pmjSTFoM8Ig4twccfckVqXFHWbvvaezEN3tBQwvTz3SBtCkUP1A==; 25:TZadMu1vzuSyQyHqrEZ4kuPsNeI6Sgaa0Fm19J7S9vGFm8+IkTX1Av+EMGvEeFtRtFE0dcokcoEdFvGsEzhVIexwOPoC4G3oJO0QRBwwMgL2lqgO7x78Ez5uKrWTa9v0Y56vROGex4f76vrdDSpiMWJ1CBP48JGZhJSncP/e+TfCsQI5Q4SAQV61TU/sk46k2FEY4OStE4U8TtS0IFOm9Rkzs0hWnDiWlwsi/2IpJD6MoGX4/QYPA/exgVMsbhHUW+1cSW2UbAyGmFQNFqbveDGcpA8jDGWTT8pUt7i/Nz4YduOsmHCVHwUN0I0N6LW4q0jQef/2NI3DmTHxD2uwTw== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:BLUPR0301MB1540; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 20:TztFL3ECnvk8h05f4nG4adDZnerx5rWM5Gk4kHCE8lDJsArnq2zAWGNWLwLmQmQYyIrWjHkUQ//HvRZUdTHO0hZO6yiQNKgP05Il41wZk0RiaK06hghnAbIyC270qW0o8WL5A0lEM5QDMK7htD/MqRIQ6xrqA7UjOsmSDrvP9U9QvIZRcLgZA8+LkNLRMKlceuLNNzNFtodguD0R8d1r55DHRPPiVWHEJDavChiiZ8ZxfFb8u13J+ucCuHsG0Nkqr5+Dl9Ssdumi3ql5tYxXhSEamZzH+NTy1JzaqIv/JAPgdkOEspzf5Y3e+aOj09a4k1urDAfwyVz1+7ZvxCdLIlWY2zG7hjXmdq2PxmMrU4+beMylHNNerdSJE6Gw3is0J0Hgzu5NGqwpox+otqzhEeOkmpKWafec8624eHsYndORIOcbGCzI/ESQ8qGp1b2tM82528NG+igOUFKW1Jakn4dVKxM5ZAXUedXarwQlZ7nKZmrvjSWJMY8aL6eSrb+6ro1ExHvQeJsMWAfRS0FuX/K6pgKcws96BsRimTEYgqpNbegF25UITHBJa30RVgjTx6x7NZuKa2Q4KwzV2XBWnhQOq1EtGTtD1af77G71WzU= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(61425038)(601004)(2401047)(5005006)(13016025)(8121501046)(13018025)(10201501046)(3002001)(61426038)(61427038); SRVR:BLUPR0301MB1540; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0301MB1540; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 4:f36l2FIMlY0gmFvIU/4/RImzjeCOgrW58HHFWCS0ClQfS/PKBZF7lnYLRT8GGBEP/P/zKP6XNhixcwKEYVmt+Squ+9sr1hxwcQfcFNkdWOTrEmaIjgcxuyH8RQErj/Se77KnVcvWdU66TXk3EqrVkQRTj0fcAlAIZgpQm/zYSD/iShmMc0fHb93SNP5Mw0YrIQ3Rk4XgebsxHhET9wTziRPpg6GmP1GJGdIShju+MzFRAFw/WTrCGIoyq56mFOmqd6Mhg5uoaMW1WKa4LC0qgLxqGqujJpOQEG0OJujparwNcx1oaECnB1hEMmsHC7C64vqHbB9fM7Gy16Nmp5Yd0eieQusincmrHHn0Hq15ckOjb9lSx4aYdd+PZC5VBhvuzGw/vl1gqg3Te3B7jNUbRy24/3IQIe6VIxBYLz80T/D9+d/NBc4WQls8FMTJH8SJTcaWzHYpOclXGi91glPY5Q== X-Forefront-PRVS: 084285FC5C X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0301MB1540; 23:7DVRo5D31+pBpdQgTaqjuJd5ZScbYpVuyromR38?= =?us-ascii?Q?MuQ2X4rCNgulBPoTUwV27n1OABmzNKb2C35KJfDhg3tEkJE0AKTse4UxecN0?= =?us-ascii?Q?UUEH4TBf8aD/+4gTJpsSz6G6Y1SaEpF0bIeFOnVec9TGCr7hYvZy+elf5op/?= =?us-ascii?Q?n+MpizpUG2QW8B9NmKEoO0tHcxx79p/4ZiprSqZQqScqRvFciGTfRHIZaHUG?= =?us-ascii?Q?UNCZvktZvGWjIww0LG2BzgWT2zFsdUUHR+7JyoWHTUTz8yFrtbIW8ZNgD6Ep?= =?us-ascii?Q?D3ET9YOOihhu1moE3luD+H6jlj6+cG4OodMH+OcAyxDqPOLXu0iuflfOxY3Y?= =?us-ascii?Q?mYMcPSJlQaOpkLln+OhPOHugUfrDkOBpnf9Bay6+O6in59YxVEh3rfJWO5vX?= =?us-ascii?Q?VptNAT9POb0wXlwaa1A2f2OAIb7fw4Vp/uuF6gYvRX1FHR4ybR0AfkYlAoNK?= =?us-ascii?Q?PtA7VwFdW/gq6mJN/jrvN2vyoXwh9OVGixInAFwGwrUnZLyV/OYcvIbZrfFD?= =?us-ascii?Q?p6uLQmfiyC/uNwrPkenQPjKXT7f/4HgAwJ7yS5dPazSMwLOJracwtaDdDaRL?= =?us-ascii?Q?38eAAslWepgNnjBbJu/kb5lnuoCletzHJCFJhKKzboGMa3Lz1PT+ssNpmo8l?= =?us-ascii?Q?rjJkPBTKY7M80qbkVO/94TuzMOK6yyPflZfR1zfsJEMcNa8yL8755hHkVDLV?= =?us-ascii?Q?gYvYVkOx/nBaRp9Et4DS0EwHIKePPFJUXoWCJbj4ghh8iS+UL1avv7dgaCL0?= =?us-ascii?Q?IzOK2H2JNsVssAnZi+/gP5HzMPGPh55Ot4IXbjSSJ4CETWIBDKSuIKHuBgBa?= =?us-ascii?Q?61i47BVWV/rXmQL9wAPZj2yGbGCyysuIpYfS60Buw1hgNTOzQxJeybTGDjcZ?= =?us-ascii?Q?sFsy7Xdas0B1ScObeUSkzyYDcZgi5+L6j0xG1DfmwBCnGpWgEQutQhaSng/4?= =?us-ascii?Q?pOPaZfX82qsRPaXF9AFRpGIikwMHqoku73ISLNmo7hFXH7sU/AimbmjhNVjT?= =?us-ascii?Q?3iNac3Xyw1U/4QGI2lmJrCS/lheQgKIMMFY9ImEO9U6sBzBMdLXqirOJuTmh?= =?us-ascii?Q?9WudF1Pc7Z/Pgkdt2spw4D8Y/APp7qmEzUPtkymTQn+k6JG4bcUbNMuHjrHE?= =?us-ascii?Q?2erD+lBkMnKzFXUnciSxV7vNwiQ8OTdFyiSKfrlcbyDkquOan7St3zgjpkej?= =?us-ascii?Q?o0qoMQY+ivLSflYTErJGA9qWX1UO3VZeVXIQCujjotTa94HZy7L5oIF1oY4J?= =?us-ascii?Q?yC2Uk1XOv2wPNSLH/tr/d88upGMn7uLjAHHPwoze8mfn8gPo4kbc7Or4h84G?= =?us-ascii?Q?010ljcaNa5Nl3hVZXKWpZd8Iq7QpR2QvUtJLo524Xvc88?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0301MB1540; 5:eDsOs3g22CnTop9Z/8cL1SXS8cRoBVrP6Pz7L/c3NCn1+b7r0FEpKJNP8QfRmcRWfzcj8WR64AX8SbDprAkUbGQAE0KTT/3uQEORhw3PcMhGgq9lu2PCT56URc1cx9vMHN2DtJ8dwyQNHyGFYbmA/g==; 24:jj6/YCkK79ROyDQLGNQ8t4ibspl5IgAlY0JLig/OkXBvGN5vSHdAIKV5QrrfB+YAcOM1GygPIJBeThwEoeU59G/KgAKi9k3u+zbytfrGEFA= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2016 13:35:05.6716 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[206.191.228.180]; Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0301MB1540 X-Mailman-Approved-At: Thu, 04 Feb 2016 14:21:10 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 13:50:17 -0000 Did you enable LRO on FreeBSD side (check 'ifconfig' output)? Linux default= enables GRO (see the output of 'ethtool -k eth0'). Thanks Hongjiang Zhang -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Meyer, Wolfgang Sent: Wednesday, February 3, 2016 9:37 PM To: 'freebsd-net@FreeBSD.org' Cc: 'freebsd-performance@FreeBSD.org' Subject: ixgbe: Network performance tuning (#TCP connections) Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://na01.safelinks.protection.outlook.com/= ?url=3Dhttps%3a%2f%2fcalomel.org%2ffreebsd_network_tuning.html%2c&data=3D01= %7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12= %7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DxsMoC%2b1ZcnoHBnPqhLUMDIr8V= LBcLejnrXgkRyDWzYc%3d https://na01.safelinks.protection.outlook.com/?url=3D= https%3a%2f%2fwww.freebsd.org%2fdoc%2fhandbook%2fconfigtuning-kernel-limits= .html%2c&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DXNqvrYfTN= zfe2btrip%2f5FoX3iTTpTSbNrDjbhtVBevo%3d https://na01.safelinks.protection.o= utlook.com/?url=3Dhttps%3a%2f%2fpleiades.ucsc.edu%2fhyades%2fFreeBSD_Networ= k_Tuning&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3D%2bQ66X%2= frnqNakX%2fSGcK08QTTrsDjUUWBHOXu6%2fOBIBNQ%3d) but most of them didn't have= any measuarable impact. Final sysctl.conf and loader.conf settings see bel= ow. Actually the only tunables that provided any improvement were identifie= d to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value= of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set = to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang : cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" : kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fww= w.hob.de%2fredirect%2fhob.html&data=3D01%7c01%7chonzhan%40064d.mgd.microsof= t.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47= %7c1&sdata=3D%2fenM%2fs72BvfP9H6CnrFeXqhrZoetovqoIMB%2bk0RFfQM%3d - Xing: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fw= ww.hob.de%2fredirect%2fxing.html&data=3D01%7c01%7chonzhan%40064d.mgd.micros= oft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db= 47%7c1&sdata=3DSi0aQmWV%2ba%2fWx5%2f6tZtOWQDj5cmq9t57DhA2h0qsa7Y%3d - LinkedIn: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2flinkedin.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3DBc0iX1u2twaazr%2fzi2wuTRqTZOk2rwtu61lNqJrui14%3d - HOBLink Mobile: https://na01.safelinks.protection.outlook.com/?url=3Dhttp= %3a%2f%2fwww.hob.de%2fredirect%2fhoblinkmobile.html&data=3D01%7c01%7chonzha= n%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f= 141af91ab2d7cd011db47%7c1&sdata=3DI1ViplWsQ2HMWM%2fLnhWRKseziJBoNlyVBj2wiWF= x1wM%3d - Facebook: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2ffacebook.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3D1WliqTPX5YN3AdvK6DzAB7yDkQmjyC3jh%2f47PZ2uU7Y%3d - Twitter: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2ftwitter.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DgaLO9OTaBis4F3IGoFY55nwMIOGPaZ0ri%2fK7N7nx7kI%3d - YouTube: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2fyoutube.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DT%2fhNogZgLHoTCOj%2fKjsSJhPDlBqyxCAD8tJj5fueiqw%3d - E-Mail: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2= fwww.hob.de%2fredirect%2fmail.html&data=3D01%7c01%7chonzhan%40064d.mgd.micr= osoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011= db47%7c1&sdata=3DQnWAw2culSVFforLRWYgXfyHaj4PyB4Hn1rj4jVQvjU%3d HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 ______= _________________________________________ freebsd-net@freebsd.org mailing list https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists.fr= eebsd.org%2fmailman%2flistinfo%2ffreebsd-net&data=3D01%7c01%7chonzhan%40064= d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91= ab2d7cd011db47%7c1&sdata=3DGU28hzKU5SSyA3hbP1PNtUNh7G5ut%2fMD6mdRmuJNkEs%3d To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Thu Feb 4 13:50:48 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C128A9C705; Thu, 4 Feb 2016 13:50:48 +0000 (UTC) (envelope-from honzhan@microsoft.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bn0107.outbound.protection.outlook.com [157.56.110.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 93C142C3; Thu, 4 Feb 2016 13:50:47 +0000 (UTC) (envelope-from honzhan@microsoft.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=7XURdZm9tz1byzix476KT1rU4mcxIKyGIjlUxC2DDXM=; b=HkoVMy5EBY791q+3lu0JoTmu9DZ7jGwjmWZmSPTTlUCmW1ziKIQGJEI2b1Pb4l9DoI8EO4DUrwR+dvStOqn1dp2vGDGelCyQ2ETx4RCdaNTq8442dfoHumhGIHV4mLhVg+w5td0LcZlZqfm4V8qHBsCYVt4b75YVdofMfEDpN7o= Received: from CH1PR03CA003.namprd03.prod.outlook.com (10.255.156.148) by CY1PR0301MB2089.namprd03.prod.outlook.com (10.164.2.147) with Microsoft SMTP Server (TLS) id 15.1.396.15; Thu, 4 Feb 2016 13:35:15 +0000 Received: from BN1BFFO11FD014.protection.gbl (10.255.156.132) by CH1PR03CA003.outlook.office365.com (10.255.156.148) with Microsoft SMTP Server (TLS) id 15.1.396.15 via Frontend Transport; Thu, 4 Feb 2016 13:35:15 +0000 Authentication-Results: spf=pass (sender IP is 206.191.228.180) smtp.mailfrom=microsoft.com; hob.de; dkim=none (message not signed) header.d=none;hob.de; dmarc=pass action=none header.from=microsoft.com; Received-SPF: Pass (protection.outlook.com: domain of microsoft.com designates 206.191.228.180 as permitted sender) receiver=protection.outlook.com; client-ip=206.191.228.180; helo=064-smtp-out.microsoft.com; Received: from 064-smtp-out.microsoft.com (206.191.228.180) by BN1BFFO11FD014.mail.protection.outlook.com (10.58.144.77) with Microsoft SMTP Server (TLS) id 15.1.409.7 via Frontend Transport; Thu, 4 Feb 2016 13:35:14 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net (141.251.56.18) by SG2PR3002MB0105.064d.mgd.msft.net (141.251.56.17) with Microsoft SMTP Server (TLS) id 15.1.403.10; Thu, 4 Feb 2016 13:35:10 +0000 Received: from SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) by SG2PR3002MB0106.064d.mgd.msft.net ([141.251.56.18]) with mapi id 15.01.0403.016; Thu, 4 Feb 2016 13:35:10 +0000 From: Hongjiang Zhang To: "Meyer, Wolfgang" , "'freebsd-net@FreeBSD.org'" CC: "'freebsd-performance@FreeBSD.org'" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAHKtgg Date: Thu, 4 Feb 2016 13:35:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [180.170.80.203] X-MS-Office365-Filtering-Correlation-Id: 178e7cc1-5c93-4225-26e5-08d32d680392 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD014; 1:ntAtAWEK8GlRhOfrLvWWo0iNr05s703QjpHSg8sEVBLHjr4e57tdXWwKPZnyBOu4IAWJafgRZKXA+Kc0lgPysd0SAlKWHJqtX1IEP/XuoaMpZUkyHWCEIB+GssjYU802kAPLGsBjpF9ngaT+Z7EftlMGMb5LG0TXI0hp9kxkWr4m4skRjqH3C7fTmiAS009xo26h/jQ5Xl83swrN3pTUK4GgKEjIHUPJ7SMVAc3CGW8uyKoznTSnhM36ltQwfAnqLyzBZIFIaT6qElRfXZM+RuMqrabDN/J1icxjL62q26I5uqD5n0VodV/w/8sOdD4ehPTq3ZwHxgYuxSqgRRdtGg== X-Forefront-Antispam-Report: CIP:206.191.228.180; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(438002)(377454003)(3905003)(13464003)(199003)(189002)(24736003)(15975445007)(2950100001)(6116002)(97756001)(33646002)(3846002)(102836003)(47776003)(23726003)(2900100001)(92566002)(87936001)(586003)(5001960100002)(5008740100001)(76176999)(66066001)(86146001)(6806005)(1096002)(54356999)(189998001)(11100500001)(1220700001)(16796002)(50986999)(108616004)(4326007)(19580395003)(2906002)(86612001)(5003600100002)(575784001)(86362001)(10400500002)(19580405001)(5005710100001)(50466002)(46406003)(10290500002)(5001770100001)(106466001)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0301MB2089; H:064-smtp-out.microsoft.com; FPR:; SPF:Pass; MLV:sfv; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 2:yzQobEZ8/kdGTNoEndrvh42hCAgw9OjqojMd9No6JDoETWyEzXN5cN6viYGJMNvZWS52dNmqQVwoD9QYjDec/owBLUyhhoIlwYJjWZEi3Y+sT7LTcOxNc/bQ2J9nEhvKs1mIcBNYgqoPZp+USnlWDjFBySR1DyHPRzYb8umCI26voFP1J3Uq2V4YaPM1p2Cv; 3:8UsR44mbp2hsE1Oe9yacrhd3jnqM4Qm5tXT6RMyP2pJZhHiKFx6Wz5dza5snSh1vN8BmK0ECPa7s2ujbi2WAYsv0zwX0Z8VXcBXnV4Pysgn7fiMw7M9GtFMs+QWaEcrmvqP9oKhzmYzcWXRuuZTmhwtJtIVU3ILvtXjvrlND5eI6l1La6yRLziaaCyfeF3PeJp4UqfxhHQzjCdRkiON3OVThaRpsRPvlPZ5l+7yXAYQfeJmKMECe1zFUW0tdI5dW9arDkGv4+7sDG6czouUc8w==; 25:Y2SPeu6ph74MT6oSz9FllACCD5AA3OtY3CXwnNlfJA9lTKo+Xzh7oeXqhJLBh1kIiJWJs6XOipwMEUqypeQPAUNBxTIIwu5npgeFSbMpDNBGlmwZsq0l8kXx8cbky69im27sI5CkevRyjKyCS/7iKK4o+ZcAJ3d/p5LT5WLSDjQfE+T1liLchylh+bCkjXO8JC4Dk0kTEDDL60ZCIDpBAvGqbg4OJv3S+VX57uBS9HpMX/peIgGWEq/SAGw8a9htcCeEYr9PNl9VtJ2qyKIevglClKuFWQu1hHzv8nf7Cpb3Y18Q4Rwc+mASqwQLALCN0Z4fPrCfuo2DiHwHNXAzdA== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501002); SRVR:CY1PR0301MB2089; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 20:RDCqWzyvNhSg+JxrvoKeHN/kuNwg+r4IzkBstIZFYdxaELRXVHTa5cDcSy0z+f6/P35cOeEpH6wcObwQETOhkK9XRhTotWl+ZS+zlcKtfP0kRt6loATW6QTllcdR6OQnqeTjdyjSIeiKC8DD0yb0DqyUo1qT7Xs3VT0mq4qQZCRFLlbtdE3cbQl81cP/ByAkF7HIM6ufzmBMXcos/w91I8LmyjFJiOOz2PjC0zISL6hrNbORDKKp80ZsyC9zvJ4OLsoj6PTdgwftrrrdNULkZ+k3GC2el+J3gVj1jiH7i+chrou8oZBoPOdyLyk+bzLkbRqcEozC1pp+J40fDm/xlwbYCG8S0X9k5erHEwPXrTyMhDftT3MOzbQvse6WLEw72BxOA1CWQ2Iq35KjHj481cAmkkhWMqXGtPdh4aNtWwuv/2jBZ9G0MovZY7szHXo6aIWtjnmlm175BbI+lL09bSWqzg0slvYIAuodjJSjb7wn4wYZH/a9RFGgiK4lerIUmsIfcZooK+MMJSFqXSJWW2hMql1PSFq2WSLQtZB2rWrmyN9h34PcHgzYldLD+idgzi3QvbPz5LK0UGQuZQxknhKtB1yNEZGJNOUd3aW+0e0= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(61425038)(601004)(2401047)(5005006)(13016025)(8121501046)(13018025)(3002001)(10201501046)(61426038)(61427038); SRVR:CY1PR0301MB2089; BCL:0; PCL:0; RULEID:; SRVR:CY1PR0301MB2089; X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 4:UeVcVSthzafHtLZ93teCM2JvnOVOdqPw/VhKRdM0ePuzJ8tmA3KSNoAdiHdNYqWBOGFWFPRfaT0XqnGMmUVq3QgYVzMjLIXVUVol9mYnzK0B3H+Jsed7Y5by5aaHDUMn1SNcRihsxvvJnFUxMxFG2A3wNupBdBdF16uOW7KLGfWf3tNUcyGffm+1qdfNvnianiSIdE8HWc/y5A548BxLxqEmR8GJmj7SF3qMbdlHGXwXDN57QU1tRx93/G8fffJg2AAjT3JbqAcqhyTWTut5+kdf6ZWVODhz09AbOc+DF1R3xoATuy6ddz0w8ThLESEkWXabFd4rgvrBqT1vxWM7fPjhAu9fhv/+CAbwBeyF9z8f+35h30j9Lfzkb1bf6tE5ORX7xBCYkWaiSJk0msFDhwc/FoUm37dXIbzLlyaHvbJkuru+EcRjGoBhKKdOBXLnH4OamZ/XPzG5U0x9hAPF4g== X-Forefront-PRVS: 084285FC5C X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR0301MB2089; 23:wvyZaz/+/NibWBM7DEl1u2pCBdzXgEJ9mIGWjVu?= =?us-ascii?Q?wIPmDK5dpxOJ4xUKtNEgPLpZTAn7N7naBpUsyxDzFddfc18qhvYb20FdKRb2?= =?us-ascii?Q?MMcKjvXElHYlasrRnH8l1a2tba+IJrkrTjCKPNCV+FsWxCSAArfKxRZrkP8V?= =?us-ascii?Q?6xYQTP2/niSkQsnOtPDAVGaBzcHSCohWHc0oBGTEpOHrREpuChFVAClv7Rbi?= =?us-ascii?Q?3fl8W5swdLLSZm79213eZpMRJayD/bhLePM6Oqd0zy0r4baMUME20+616PIC?= =?us-ascii?Q?8ZEKGmQX9AaoxAXvLxHiCm2VqV00IZjL6+ShqIapzae4Ka/3DXakIU9OiDdN?= =?us-ascii?Q?QkJn9f5ZzwlAZCOaz/6msyJAh9HWupyND02OCSjqaRgKJ2IhEYO+3W9uwq30?= =?us-ascii?Q?LQXYY1Dfp6OOnIMy8+awXNodLHcFNQN4KTm2GwAeGb/MqIM7QOWuACon9HFS?= =?us-ascii?Q?S33pLoNec/tKqUiZyelR5DhdKzp2DRfZem6CnQJgUgRnpVnE64IkSFNyTOCG?= =?us-ascii?Q?a/HAQ0d6yBm++eRJHwHoSu1UMcBLBhE5qjfQufv0/KrstaYRlgyqcwMWtqHG?= =?us-ascii?Q?sRScFkESOLGcTbHsVlJZfLQ0ODx2bUgaDSy9oZbzQA5P+ekfUBXdq8CCI9m3?= =?us-ascii?Q?YzeuMFyvcSBzh0Z1A6bF8TZ4QmmGX/HYchRWlJwp8GCUjSJuPW92VnhV/F+y?= =?us-ascii?Q?6SV37hE3eEw89cO1cmaf663ZUTbtjmed2Gmtl8gy17Zr1TCs5OKCNq7K7bYZ?= =?us-ascii?Q?sxE+93I2t+gOWJxx05RDSRo7JfaxvMn3EyCDQHZPxyGj5kD21R2o/m6ixBvT?= =?us-ascii?Q?QL2mSYd8RzcCPxF14p0T02uO3sWE30KgfbkXO1WgxsqQMQ8Jt7jOJtrOHAlE?= =?us-ascii?Q?vzkn3GSipYy6C5lFn4FEn68ZgYkqc6H2LtZnHAMs10iPzkC13YENQzu/Qcel?= =?us-ascii?Q?5I/leoCr6QXPc27V6HiZPUY0MzwbeHK8eekZARK45AbF+SGLEeqWftxs1F5h?= =?us-ascii?Q?JkIqkpctTcEuUrnLJIcwYrNzc/QxMt1Np2UvaHNVZ3CKapE6WzqHW4OUZmZq?= =?us-ascii?Q?TqRMVg6Jqmfs7AAWkWQco1boz1GiH639h0UYiHR0VtCaTvEGzHOSDozyXBlz?= =?us-ascii?Q?oBjA/OSUsYm1kaVJoSk1t88u6JTfQfNyiZqeoosqYxH2vcfYvBfwYyU7v01G?= =?us-ascii?Q?n9U0SFIofLzkdSNgTr/w6dziwV0lGFKzkGGTqv4lHHvQwX+ZlMWICYg3Ws+P?= =?us-ascii?Q?E6HFdM5rRvQButcVBwnvfIbdb8Dp5OethJclbaVdi5ehsPiWyBrZmpZJOkvC?= =?us-ascii?Q?0rrZyDvxp1wmod04DGctCVc2SHAkQzZ6la/f8iBCNrN+E?= X-Microsoft-Exchange-Diagnostics: 1; CY1PR0301MB2089; 5:MEFAOzt/1NyF4xEh1bB9ouM0AptaVEQijN2ngECnXT6oyBZ8nRXs5Xux4OF75JrF8s/SpgWCGx7CFrjr4L/AA5Xj2gMwKryqJkPrvuB2iYI8/WO23rFsTAiQbI0nPW9iONFV4sH4mIyPXeJ/zOIwVA==; 24:BhMQ+p0Jz9GLDZy9zSIv2ko53fI2S30viC6Vpx5DrIFu6U6R8SWZHVx1aOMJbWo1Hoe29K8d0wIPxhydVA9J7aaxi4lTzY+nB8cQRxue8+c= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2016 13:35:14.4450 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47; Ip=[206.191.228.180]; Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0301MB2089 X-Mailman-Approved-At: Thu, 04 Feb 2016 14:33:20 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 13:50:48 -0000 Please check whether LRO is enabled on your FreeBSD server with "ifconfig".= Linux default enables GRO (see the output of 'ethtool -k eth0'), which cov= ers LRO optimization. Thanks Hongjiang Zhang -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Meyer, Wolfgang Sent: Wednesday, February 3, 2016 9:37 PM To: 'freebsd-net@FreeBSD.org' Cc: 'freebsd-performance@FreeBSD.org' Subject: ixgbe: Network performance tuning (#TCP connections) Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://na01.safelinks.protection.outlook.com/= ?url=3Dhttps%3a%2f%2fcalomel.org%2ffreebsd_network_tuning.html%2c&data=3D01= %7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12= %7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DxsMoC%2b1ZcnoHBnPqhLUMDIr8V= LBcLejnrXgkRyDWzYc%3d https://na01.safelinks.protection.outlook.com/?url=3D= https%3a%2f%2fwww.freebsd.org%2fdoc%2fhandbook%2fconfigtuning-kernel-limits= .html%2c&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DXNqvrYfTN= zfe2btrip%2f5FoX3iTTpTSbNrDjbhtVBevo%3d https://na01.safelinks.protection.o= utlook.com/?url=3Dhttps%3a%2f%2fpleiades.ucsc.edu%2fhyades%2fFreeBSD_Networ= k_Tuning&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3D%2bQ66X%2= frnqNakX%2fSGcK08QTTrsDjUUWBHOXu6%2fOBIBNQ%3d) but most of them didn't have= any measuarable impact. Final sysctl.conf and loader.conf settings see bel= ow. Actually the only tunables that provided any improvement were identifie= d to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value= of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set = to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang : cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" : kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fww= w.hob.de%2fredirect%2fhob.html&data=3D01%7c01%7chonzhan%40064d.mgd.microsof= t.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47= %7c1&sdata=3D%2fenM%2fs72BvfP9H6CnrFeXqhrZoetovqoIMB%2bk0RFfQM%3d - Xing: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fw= ww.hob.de%2fredirect%2fxing.html&data=3D01%7c01%7chonzhan%40064d.mgd.micros= oft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db= 47%7c1&sdata=3DSi0aQmWV%2ba%2fWx5%2f6tZtOWQDj5cmq9t57DhA2h0qsa7Y%3d - LinkedIn: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2flinkedin.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3DBc0iX1u2twaazr%2fzi2wuTRqTZOk2rwtu61lNqJrui14%3d - HOBLink Mobile: https://na01.safelinks.protection.outlook.com/?url=3Dhttp= %3a%2f%2fwww.hob.de%2fredirect%2fhoblinkmobile.html&data=3D01%7c01%7chonzha= n%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f= 141af91ab2d7cd011db47%7c1&sdata=3DI1ViplWsQ2HMWM%2fLnhWRKseziJBoNlyVBj2wiWF= x1wM%3d - Facebook: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2ffacebook.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3D1WliqTPX5YN3AdvK6DzAB7yDkQmjyC3jh%2f47PZ2uU7Y%3d - Twitter: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2ftwitter.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DgaLO9OTaBis4F3IGoFY55nwMIOGPaZ0ri%2fK7N7nx7kI%3d - YouTube: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2fyoutube.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DT%2fhNogZgLHoTCOj%2fKjsSJhPDlBqyxCAD8tJj5fueiqw%3d - E-Mail: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2= fwww.hob.de%2fredirect%2fmail.html&data=3D01%7c01%7chonzhan%40064d.mgd.micr= osoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011= db47%7c1&sdata=3DQnWAw2culSVFforLRWYgXfyHaj4PyB4Hn1rj4jVQvjU%3d HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 ______= _________________________________________ freebsd-net@freebsd.org mailing list https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists.fr= eebsd.org%2fmailman%2flistinfo%2ffreebsd-net&data=3D01%7c01%7chonzhan%40064= d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91= ab2d7cd011db47%7c1&sdata=3DGU28hzKU5SSyA3hbP1PNtUNh7G5ut%2fMD6mdRmuJNkEs%3d To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Fri Feb 5 18:05:46 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA39FA9D178 for ; Fri, 5 Feb 2016 18:05:46 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from hobex19.hob.de (hobex19.hob.de [212.185.199.31]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hobex19", Issuer "hobex19" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3022B817; Fri, 5 Feb 2016 18:05:45 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from HOBEX22.hob.de (172.22.1.4) by hobex19.hob.de (172.25.1.31) with Microsoft SMTP Server (TLS) id 14.2.347.0; Fri, 5 Feb 2016 19:05:11 +0100 Received: from HOBEX11.hob.de ([fe80::b99f:1c54:7122:49b4]) by HOBEX22.hob.de ([::1]) with mapi id 14.02.0387.000; Fri, 5 Feb 2016 19:05:32 +0100 From: "Meyer, Wolfgang" To: "'K. Macy'" , Allan Jude CC: "freebsd-performance@freebsd.org" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAJFIMAAAN+hIAAYzfxMA== Date: Fri, 5 Feb 2016 18:05:32 +0000 Message-ID: References: <56B23DEA.1060307@freebsd.org> In-Reply-To: Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.24.71.140] old-x-esetresult: clean, is OK old-x-esetid: 46BD7B39450636371DF120 x-esetresult: clean, is OK x-esetid: 46BD7B39450636371DF120 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 18:05:46 -0000 > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- > performance@freebsd.org] On Behalf Of K. Macy > Sent: Mittwoch, 3. Februar 2016 20:31 > To: Allan Jude > Cc: freebsd-performance@freebsd.org > Subject: Re: ixgbe: Network performance tuning (#TCP connections) > > Also - check for txq overruns and rxq drops in sysctl. 64 is very low on > FreeBSD. You may also look in to increasing the size of your pcb hash tab= le. > > > > On Wed, Feb 3, 2016 at 9:50 AM, Allan Jude wrote: > > On 2016-02-03 08:37, Meyer, Wolfgang wrote: > >> Hello, > >> > >> we are evaluating network performance on a DELL-Server (PowerEdge > R930 with 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) > with 10 GbE-Cards. We use programs that on server side accepts connection= s > on a IP-address+port from the client side and after establishing the > connection data is sent in turns between server and client in a predefine= d > pattern (server side sends more data than client side) with sleeps in > between the send phases. The test set-up is chosen in such way that every > client process initiates 500 connections handled in threads and on the se= rver > side each process representing an IP/Port pair also handles 500 connectio= ns > in threads. > >> > >> The number of connections is then increased and the overall network > througput is observed using nload. On FreeBSD (on server side) roughly at > 50,000 connections errors begin to occur and the overall throughput won't > increase further with more connections. With Linux on the server side it = is > possible to establish more than 120,000 connections and at 50,000 > connections the overall throughput ist double that of FreeBSD with the sa= me > sending pattern. Furthermore system load on FreeBSD is much higher with > 50 % system usage on each core and 80 % interrupt usage on the 8 cores > handling the interrupt queues for the NIC. In comparison Linux has <10 % > system usage, <10 % user usage and about 15 % interrupt usage on the 16 > cores handling the network interrupts for 50,000 connections. > >> > >> Varying the numbers for the NIC interrupt queues won't change the > performance (rather worsens the situation). Disabling Hyperthreading > (utilising 40 cores) degrades the performance. Increasing MAXCPU to utili= se > all 80 cores won't improve compared to 64 cores, atkbd and uart had to be > disabled to avoid kernel panics with increased MAXCPU (thanks to Andre > Oppermann for investigating this). Initiallly the tests were made on 10.2 > Release, later I switched to 10 Stable (later with ixgbe driver version 3= .1.0) > but that didn't change the numbers. > >> > >> Some sysctl configurables were modified along the network performance > guidelines found on the net (e.g. > https://calomel.org/freebsd_network_tuning.html, > https://www.freebsd.org/doc/handbook/configtuning-kernel-limits.html, > https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning) but most of > them didn't have any measuarable impact. Final sysctl.conf and loader.con= f > settings see below. Actually the only tunables that provided any > improvement were identified to be hw.ix.txd, and hw.ix.rxd that were > reduced (!) to the minimum value of 64 and hw.ix.tx_process_limit and > hw.ix.rx_process_limit that were set to -1. > >> > >> Any ideas what tunables might be changed to get a higher number of TCP > connections (it's not a question of the overall throughput as changing th= e > sending pattern allows me to fully utilise the 10Gb bandwidth)? How can I > determine where the kernel is spending its time that causes the high CPU > load? Any pointers are highly appreciated, I can't believe that there is = such a > blatant difference in network performance compared to Linux. > >> > >> Regards, > >> Wolfgang > >> > > > > I wonder if this might be NUMA related. Specifically, it might help to > > make sure that the 8 CPU cores that the NIC queues are pinned to, are > > on the same CPU that is backing the PCI-E slot that the NIC is in. > > > > > > -- > > Allan Jude > > As I was telling in my original message, the rxd and txd values were more o= r less the only ones that changed my numbers to the better when reducing th= em. Not that I understood that behaviour but a double-check now revealed th= at I stand corrected on this observation. Raising the value (to 1024) now d= id not only degrade througput to my original bad numbers but to the opposit= e slightly improved it (but only barely measurable compared to measurement = variation). Don't know what cross interaction was leading to my original ob= servation. Concerning pcb hash table size I only found net.inet.sctp.pcbhashsize and t= hat had no influence. Not sure whether sctp plays a role at all in my probl= em. Regards, Wolfgang Meyer ________________________________ Follow HOB: - HOB: http://www.hob.de/redirect/hob.html - Xing: http://www.hob.de/redirect/xing.html - LinkedIn: http://www.hob.de/redirect/linkedin.html - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html - Facebook: http://www.hob.de/redirect/facebook.html - Twitter: http://www.hob.de/redirect/twitter.html - YouTube: http://www.hob.de/redirect/youtube.html - E-Mail: http://www.hob.de/redirect/mail.html HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 From owner-freebsd-performance@freebsd.org Fri Feb 5 18:16:11 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0CCE7A9D534 for ; Fri, 5 Feb 2016 18:16:11 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from hobex19.hob.de (hobex29.hob.de [212.185.199.32]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9EF331200; Fri, 5 Feb 2016 18:16:10 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from HOBEX12.hob.de (172.22.1.12) by hobex29.hob.de (172.25.1.32) with Microsoft SMTP Server (TLS) id 14.2.347.0; Fri, 5 Feb 2016 19:15:38 +0100 Received: from HOBEX11.hob.de ([fe80::b99f:1c54:7122:49b4]) by HOBEX12.hob.de ([::1]) with mapi id 14.02.0387.000; Fri, 5 Feb 2016 19:15:53 +0100 From: "Meyer, Wolfgang" To: 'Allan Jude' , "freebsd-performance@freebsd.org" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAQ4pyAAACKbgAAXtwogA== Date: Fri, 5 Feb 2016 18:15:52 +0000 Message-ID: References: <56B275EC.2060604@freebsd.org> In-Reply-To: <56B275EC.2060604@freebsd.org> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.24.71.140] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 18:16:11 -0000 > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- > performance@freebsd.org] On Behalf Of Allan Jude > Sent: Mittwoch, 3. Februar 2016 22:50 > To: freebsd-performance@freebsd.org > Subject: Re: ixgbe: Network performance tuning (#TCP connections) > > On 2016-02-03 16:34, Adrian Chadd wrote: > > hi, > > > > can you share your testing program source? > > > > > > -a > > > > I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel X540-AT2 tha= t I > can try to replicate this one to help debug it. > > -- > Allan Jude I'll try to do some polishing and removal of cruft next week than I hope I = will feel comfortable putting it to the public :-) Not that they are overly sophisticated programs, just some test set-up crea= ted in the past and over time one gets used to use it as a sort of private = "benchmark". Regards, Wolfgang Meyer ________________________________ Follow HOB: - HOB: http://www.hob.de/redirect/hob.html - Xing: http://www.hob.de/redirect/xing.html - LinkedIn: http://www.hob.de/redirect/linkedin.html - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html - Facebook: http://www.hob.de/redirect/facebook.html - Twitter: http://www.hob.de/redirect/twitter.html - YouTube: http://www.hob.de/redirect/youtube.html - E-Mail: http://www.hob.de/redirect/mail.html HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 From owner-freebsd-performance@freebsd.org Fri Feb 5 18:19:43 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06E46A9D753 for ; Fri, 5 Feb 2016 18:19:43 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id DEA681AA7 for ; Fri, 5 Feb 2016 18:19:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [192.168.1.10] (unknown [192.168.1.10]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 401F5DFED for ; Fri, 5 Feb 2016 18:19:36 +0000 (UTC) Subject: Re: ixgbe: Network performance tuning (#TCP connections) To: freebsd-performance@freebsd.org References: <56B23DEA.1060307@freebsd.org> From: Allan Jude Message-ID: <56B4E7B0.9010509@freebsd.org> Date: Fri, 5 Feb 2016 13:19:28 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 18:19:43 -0000 On 2016-02-05 13:05, Meyer, Wolfgang wrote: > > > As I was telling in my original message, the rxd and txd values were more or less the only ones that changed my numbers to the better when reducing them. Not that I understood that behaviour but a double-check now revealed that I stand corrected on this observation. Raising the value (to 1024) now did not only degrade througput to my original bad numbers but to the opposite slightly improved it (but only barely measurable compared to measurement variation). Don't know what cross interaction was leading to my original observation. > > Concerning pcb hash table size I only found net.inet.sctp.pcbhashsize and that had no influence. Not sure whether sctp plays a role at all in my problem. > > Regards, > Wolfgang Meyer > > I think the one you are looking for is: net.inet.tcp.tcbhashsize See if doubling that makes a difference. -- Allan Jude From owner-freebsd-performance@freebsd.org Fri Feb 5 18:52:10 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9101A76A5C for ; Fri, 5 Feb 2016 18:52:10 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from hobex19.hob.de (hobex19.hob.de [212.185.199.31]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hobex19", Issuer "hobex19" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DCAD984; Fri, 5 Feb 2016 18:52:09 +0000 (UTC) (envelope-from wolfgang.meyer@hob.de) Received: from HOBEX22.hob.de (172.22.1.4) by hobex19.hob.de (172.25.1.31) with Microsoft SMTP Server (TLS) id 14.2.347.0; Fri, 5 Feb 2016 19:51:42 +0100 Received: from HOBEX11.hob.de ([fe80::b99f:1c54:7122:49b4]) by HOBEX22.hob.de ([::1]) with mapi id 14.02.0387.000; Fri, 5 Feb 2016 19:52:07 +0100 From: "Meyer, Wolfgang" To: 'Allan Jude' , "freebsd-performance@freebsd.org" Subject: RE: ixgbe: Network performance tuning (#TCP connections) Thread-Topic: ixgbe: Network performance tuning (#TCP connections) Thread-Index: AdFefq3DO+J8ScSLR4uotZFUcfP67wAJFIMAAAN+hIAAYzfxMP//9wkA///nTnA= Date: Fri, 5 Feb 2016 18:52:07 +0000 Message-ID: References: <56B23DEA.1060307@freebsd.org> <56B4E7B0.9010509@freebsd.org> In-Reply-To: <56B4E7B0.9010509@freebsd.org> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.24.71.140] old-x-esetresult: clean, is OK old-x-esetid: 46BD7B39450636371DF120 x-esetresult: clean, is OK x-esetid: 46BD7B39450636371DF120 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 18:52:10 -0000 > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- > performance@freebsd.org] On Behalf Of Allan Jude > Sent: Freitag, 5. Februar 2016 19:19 > To: freebsd-performance@freebsd.org > Subject: Re: ixgbe: Network performance tuning (#TCP connections) > > On 2016-02-05 13:05, Meyer, Wolfgang wrote: > > > > > > > As I was telling in my original message, the rxd and txd values were mo= re or > less the only ones that changed my numbers to the better when reducing > them. Not that I understood that behaviour but a double-check now > revealed that I stand corrected on this observation. Raising the value (t= o > 1024) now did not only degrade througput to my original bad numbers but t= o > the opposite slightly improved it (but only barely measurable compared to > measurement variation). Don't know what cross interaction was leading to > my original observation. > > > > Concerning pcb hash table size I only found net.inet.sctp.pcbhashsize a= nd > that had no influence. Not sure whether sctp plays a role at all in my > problem. > > > > Regards, > > Wolfgang Meyer > > > > > > I think the one you are looking for is: net.inet.tcp.tcbhashsize > > See if doubling that makes a difference. > > -- > Allan Jude Doesnt't seem to make a difference. Anyway as now system load of the cores = handling the interrupt queues seem to max out, I'll probably have to look g= iving some relief to this parameter. On Linux there was also some manual se= tting of processor affinities for the interrupt queues necessary. Regards, Wolfgang ________________________________ Follow HOB: - HOB: http://www.hob.de/redirect/hob.html - Xing: http://www.hob.de/redirect/xing.html - LinkedIn: http://www.hob.de/redirect/linkedin.html - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html - Facebook: http://www.hob.de/redirect/facebook.html - Twitter: http://www.hob.de/redirect/twitter.html - YouTube: http://www.hob.de/redirect/youtube.html - E-Mail: http://www.hob.de/redirect/mail.html HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 From owner-freebsd-performance@freebsd.org Fri Feb 5 22:57:28 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 439B2A7727C for ; Fri, 5 Feb 2016 22:57:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10A9EC44; Fri, 5 Feb 2016 22:57:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x233.google.com with SMTP id 5so24326433igt.0; Fri, 05 Feb 2016 14:57:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=lTnlnY0fjbAjME/EX6Q0mYePY6iyNRgO9jKdJ1eN5wU=; b=eRDFYHmgulIt9bkoHFa6Vr8m1aZdeaQ90Yeqth81ruwrQcGGPSSkjFye9y1vuB9P12 iNyF4IGTdTbYc0giLV4nLfyM5DJdWkdPm+VlpO+BXd6WYsVWMypBH5AciTHZPFJmP4P9 op04lu67eq0WZjP9EOHgeP5XMrJqcdoJe4YuTKfqQt9VOqku/iF2UbI4hZPN2BaWeG1c azBK639H+bYDQvn002r9PQCD+U8qbqt0HWCRcfKxI5AmKjOLq2kK6N1YfX5YoavptAIJ ppKIPu0LTZQqY45L/jldjJTtwHCY7PsNJRzdfdtWcsYgNtiZUHUiyeOtgoajnIoVssC8 rh8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=lTnlnY0fjbAjME/EX6Q0mYePY6iyNRgO9jKdJ1eN5wU=; b=fvieiwdiiSvg8KPrW+xnVzdFtoE6xJC6eVgnGfcVv2p2gco/NqwWfWmsmSEkjGWY1T 1VVVPsOFrrxdL/MDzibvl2mz0RvXxp8RuhY/1+tvdnr5aEvPMLOpQjQKqNLMO9D/R+0K AQJ7QZrzYUMB4eJvOO7o0JYv7V6F5QV9ulf8pEaKxw5boYH9nNNyZTnwljguvzp1e+nz zKOyVt+tp8L/8DYm8oU7qjoJfsBgNwT9Z5+UZcgqND/Q3onvDdVCGis7a98snYSomgKk DuiBtNdReoCo9Hm9U1em4stc/h36t4xIFqiPuWHK9s3EBN1wLDRuj9EEPwYXA/FOmi2c Ndhg== X-Gm-Message-State: AG10YOQ7lar85g1aeATVJtzlEO1xmIWUw7Dm344ngmo+r/2tO+ECb03SwGpK/2RB+v11BSj9I2IShXgfazDjLw== MIME-Version: 1.0 X-Received: by 10.50.178.178 with SMTP id cz18mr18322977igc.37.1454713047304; Fri, 05 Feb 2016 14:57:27 -0800 (PST) Received: by 10.36.14.19 with HTTP; Fri, 5 Feb 2016 14:57:27 -0800 (PST) In-Reply-To: References: <56B275EC.2060604@freebsd.org> Date: Fri, 5 Feb 2016 14:57:27 -0800 Message-ID: Subject: Re: ixgbe: Network performance tuning (#TCP connections) From: Adrian Chadd To: "Meyer, Wolfgang" Cc: Allan Jude , "freebsd-performance@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Fri, 05 Feb 2016 23:23:35 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Feb 2016 22:57:28 -0000 On 5 February 2016 at 10:15, Meyer, Wolfgang wrote: > > >> -----Original Message----- >> From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- >> performance@freebsd.org] On Behalf Of Allan Jude >> Sent: Mittwoch, 3. Februar 2016 22:50 >> To: freebsd-performance@freebsd.org >> Subject: Re: ixgbe: Network performance tuning (#TCP connections) >> >> On 2016-02-03 16:34, Adrian Chadd wrote: >> > hi, >> > >> > can you share your testing program source? >> > >> > >> > -a >> > >> >> I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel X540-AT2 that I >> can try to replicate this one to help debug it. >> >> -- >> Allan Jude > > I'll try to do some polishing and removal of cruft next week than I hope I will feel comfortable putting it to the public :-) > > Not that they are overly sophisticated programs, just some test set-up created in the past and over time one gets used to use it as a sort of private "benchmark". Please do - it'd be good to see what you're doing and figure out what's causing the poor behaviour. Also having more public benchmarks for testing and reproducibility is always good. :) -adrian > > Regards, > Wolfgang Meyer > > > ________________________________ > > Follow HOB: > > - HOB: http://www.hob.de/redirect/hob.html > - Xing: http://www.hob.de/redirect/xing.html > - LinkedIn: http://www.hob.de/redirect/linkedin.html > - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html > - Facebook: http://www.hob.de/redirect/facebook.html > - Twitter: http://www.hob.de/redirect/twitter.html > - YouTube: http://www.hob.de/redirect/youtube.html > - E-Mail: http://www.hob.de/redirect/mail.html > > > HOB GmbH & Co. KG > Schwadermuehlstr. 3 > D-90556 Cadolzburg > > Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic > > AG Fuerth, HRA 5180 > Steuer-Nr. 218/163/00107 > USt-ID-Nr. DE 132747002 > > Komplementaerin HOB electronic Beteiligungs GmbH > AG Fuerth, HRB 3416 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Sat Feb 6 06:50:26 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5AB4A9E8BF for ; Sat, 6 Feb 2016 06:50:26 +0000 (UTC) (envelope-from savetherbtz@gmail.com) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B16D7C15; Sat, 6 Feb 2016 06:50:26 +0000 (UTC) (envelope-from savetherbtz@gmail.com) Received: by mail-pa0-x234.google.com with SMTP id cy9so45831268pac.0; Fri, 05 Feb 2016 22:50:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=nhC3vZJ05KvbVDICzhuP1nIST4Yxur++HrJYJmQOAT0=; b=z9kZipfhiuJzOR5sLOKw9UPGaEVsPWo8Vjy4+wB9PsWw2o21BY5zSH3NT4IiU7TqTd kZHKbJHrRYFCi2fmdD7F8llPsi5dhmz912C+2iTO6mvrgRIi0bEDMnxvIQmhM3EMp4NW xIoMDvQ90uDkix98BAb+hA5LAwRVEMoCZ/VjPDgXJKGps7gSqewklhBZPCSNR/zrD9QW 45sLY8NnioUxZQkDiP+8Rm3xwlwTmffkcraC2iu8re2h8OvBwocQwteMwH9gtmCXo/OS pXDwavLAwypQtKV6slmilJM0vJ8HdYExNhpbf2vPjVLEV3nlSKbFuc/FoG65LAMlR4p2 Lh0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to; bh=nhC3vZJ05KvbVDICzhuP1nIST4Yxur++HrJYJmQOAT0=; b=XTvWWQTJBoKTFmQ6o/sqh30OX4FGcepjdk3BJ8yZlaK+7zVui5dVz9oWn6WDii58pC h55hgxMNWTn2r+nJ6KHccsyNsglVTKBZbA5Hzu+BBKuIE5cm4VKS8cM1pWTOCWwNacIO KxHpharqMTBAbNePiDCVejEorQoMQ5OKJunIFlIc/jSspZfmBAXfwr4fAqoUmKCieWlZ Vdyob9yTyWIkZJQkQFOPi54w6zpgVOqLgFaKpYIfqqP5XUn7W2m4BTph+W2PETGX0wsz PkoXdwzL4nKK8IIYbSqm4Z0B7V4E0xpnITOS7mazeHBCxRc/7ljruYDWw3DZt8dfVfL1 chrg== X-Gm-Message-State: AG10YOTKsImxaqd14zv0b05laS3ewrqOI63UhrnoEiYRj7h9dxGsPUKfjlWk47k7PwYzfA== X-Received: by 10.66.242.17 with SMTP id wm17mr25486321pac.155.1454741426095; Fri, 05 Feb 2016 22:50:26 -0800 (PST) Received: from [10.0.1.12] (c-98-248-221-187.hsd1.ca.comcast.net. [98.248.221.187]) by smtp.gmail.com with ESMTPSA id fa3sm28625923pab.45.2016.02.05.22.50.24 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 05 Feb 2016 22:50:25 -0800 (PST) Subject: Re: ixgbe: Network performance tuning (#TCP connections) Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Content-Type: multipart/signed; boundary="Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.6b2 From: Alexey Ivanov In-Reply-To: Date: Fri, 5 Feb 2016 22:50:20 -0800 Cc: "freebsd-performance@freebsd.org" , Adrian Chadd , Allan Jude Message-Id: References: <56B275EC.2060604@freebsd.org> To: "Meyer, Wolfgang" X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Feb 2016 06:50:27 -0000 --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii In Linux/Intel I would use following methodology for performance = analysis: Hardware: * turbostat Look for C/P states for cores, frequencies, number of SMIs. [1] * cpufreq-info Look for current driver, frequencies, and governor. * atop Look for interrupt distribution across cores Look for context switches, interrupts. * ethtool -S for stats, look for errors, drops, overruns, missed interrupts, etc -k for offloads, enable GRO/GSO, rss/rps/rfs/xps[0] -g for ring sizes, increase -c for interrupt coalescing Kernel: * /proc/net/softirq [2] and /proc/interrupts [3] Again, distribution, missed, delayed interrupts (optional) NUMA-affinity * perf top Look where kernel spends its time. * iptables Look if there are rules (if any) that may affect performance. * netstat -s, netstat -m Look for error counters and buffer counts * sysctl / grub So much to tweak here. Try increasing hashtable sizes, playing with = memory buffers and other limits. BSD has alternatives to most of these, e.g. perf -> hwpmc, ethtool -> = ifconfig, softirq -> netisr, menu.lst -> loader.conf In your case I suppose turning on interrupt coalescing along with = enabling hardware offloads may help. [0] Comparing mutiqueue support Linux vs FreeBSD = https://wiki.freebsd.org/201305DevSummit/NetworkReceivePerformance/Compari= ngMutiqueueSupportLinuxvsFreeBSD [1] You can pin your processor to a specific C-state: https://gist.github.com/SaveTheRbtz/f5e8d1ca7b55b6a7897b [2] You can analyze that data with: https://gist.github.com/SaveTheRbtz/172b2e2eb3cbd96b598d [3] You can set affinity with: https://gist.github.com/SaveTheRbtz/8875474 PS. Sorry for so many Linuxisms on freebsd-performance@ > On Feb 5, 2016, at 2:57 PM, Adrian Chadd = wrote: >=20 > On 5 February 2016 at 10:15, Meyer, Wolfgang = wrote: >>=20 >>=20 >>> -----Original Message----- >>> From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- >>> performance@freebsd.org] On Behalf Of Allan Jude >>> Sent: Mittwoch, 3. Februar 2016 22:50 >>> To: freebsd-performance@freebsd.org >>> Subject: Re: ixgbe: Network performance tuning (#TCP connections) >>>=20 >>> On 2016-02-03 16:34, Adrian Chadd wrote: >>>> hi, >>>>=20 >>>> can you share your testing program source? >>>>=20 >>>>=20 >>>> -a >>>>=20 >>>=20 >>> I have a Dual E5-2630 v3 (2x8x 2.40ghz (+HT)) with the Intel = X540-AT2 that I >>> can try to replicate this one to help debug it. >>>=20 >>> -- >>> Allan Jude >>=20 >> I'll try to do some polishing and removal of cruft next week than I = hope I will feel comfortable putting it to the public :-) >>=20 >> Not that they are overly sophisticated programs, just some test = set-up created in the past and over time one gets used to use it as a = sort of private "benchmark". >=20 > Please do - it'd be good to see what you're doing and figure out > what's causing the poor behaviour. >=20 > Also having more public benchmarks for testing and reproducibility is > always good. :) >=20 >=20 >=20 > -adrian >=20 >>=20 >> Regards, >> Wolfgang Meyer >>=20 >>=20 >> ________________________________ >>=20 >> Follow HOB: >>=20 >> - HOB: http://www.hob.de/redirect/hob.html >> - Xing: http://www.hob.de/redirect/xing.html >> - LinkedIn: http://www.hob.de/redirect/linkedin.html >> - HOBLink Mobile: http://www.hob.de/redirect/hoblinkmobile.html >> - Facebook: http://www.hob.de/redirect/facebook.html >> - Twitter: http://www.hob.de/redirect/twitter.html >> - YouTube: http://www.hob.de/redirect/youtube.html >> - E-Mail: http://www.hob.de/redirect/mail.html >>=20 >>=20 >> HOB GmbH & Co. KG >> Schwadermuehlstr. 3 >> D-90556 Cadolzburg >>=20 >> Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic >>=20 >> AG Fuerth, HRA 5180 >> Steuer-Nr. 218/163/00107 >> USt-ID-Nr. DE 132747002 >>=20 >> Komplementaerin HOB electronic Beteiligungs GmbH >> AG Fuerth, HRB 3416 >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJWtZevAAoJECvXQw+IBr2adcsQAJhR1eC1WOal8W+iCO0g+l/6 3SL+8yjbu1CnBkqXO+lHSKpilUbkb66TNTliynfzB/GVAmODClJN+WtJfK9P5hrV i0MUoMtF1jg5GxEZ8gXn5bZvVtZbaZjDsEiGPOPxy17b/AYF73k741/aOl6GA7BS sIMzBopImBCsS5avLm872EqKLHaXvHhElA9ivSqG5X8hLiubE0RDN0bJgP1IZTHV 9WEaef+PCVJnkCpN5RayRO0Cxj+GxEEsq6T/qFqmzZ94m3hT7ZpuXG7k8VENJXpK Wos4ntzzOud+1rc7+2nkAZ16PMhWJKYv1XF/3owRnuCWGEEy9Vsl9/crLkkuTK2C t4wds2QVOqyAQKJ7KngBcSBfpJs873rw3qFL7FQK8ccNCqCSyR5ZFQu9mH0NlH5g F2FuWYzCfi4cqDPjhxSoOobL5oovcYNcZbqGIAECXvZIrrXAmTwv43oloKD4Xh0N QBfLkz7BLiYi6tIDwehE+nJfpBREpGhnzflqwGdHs0ZMcEe8QDiudmZeZEjrTaf4 XUaCpPa2OIiM6qLHitQCRga8LCTe9cI8g17Lg6MfwT/8uJjKwcpIlWIcnLDOEuWK MgCi+LpluSfMTlPK+4OLLMvBW0CcVT/OtJd+UY/8GQLCWfxb2XbNBhJc7IbvT4M3 t0nD8C/ZZDg5yEwkjzbV =4pta -----END PGP SIGNATURE----- --Apple-Mail=_ACB17E14-EDA7-44C2-8FD0-6F36EBE3B85B--