Date: Sun, 28 Apr 2013 11:59:33 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Jack Vogel <jfvogel@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org>, =?iso-8859-1?Q?Cl=E9ment_Hermann_=28nodens=29?= <nodens2099@gmail.com> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 Message-ID: <1367175573.97340.YahooMailClassic@web121602.mail.ne1.yahoo.com> In-Reply-To: <CAFOYbcmbOL=P-KyTpxh5cEdYPynEXMadGWmO3L_enYAWe9OYzA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The point of lists is to be able to benefit from other's experiences so you don't have to waste your time "trying" things that others have already done. I'm not pontificating. I've done the tests. There's no reason for every person who is having to exact same problem to do the same tests over and over, hoping for somemagically different result. The result will always be the same. Because there's no chance of it working properly by chance. BC --- On Sun, 4/28/13, Jack Vogel <jfvogel@gmail.com> wrote: From: Jack Vogel <jfvogel@gmail.com> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 To: "Barney Cordoba" <barney_cordoba@yahoo.com> Cc: "FreeBSD Net" <freebsd-net@freebsd.org>, "Clément Hermann (nodens)" <nodens2099@gmail.com> Date: Sunday, April 28, 2013, 1:07 PM Try setting your queues to 1, run some tests, then try settingyour queues to 2, then to 4... its called tuning, and rather thanjust pontificating about it, which Barney so loves to do, you can discover what works best. I ran tests last week preparing for anew driver version and found the best results came not only whiletweaking queues, but also ring size, and I could see changes based on the buf ring size.... There are lots of things that may improve ordegrade performance depending on the workload. Jack On Sun, Apr 28, 2013 at 7:21 AM, Barney Cordoba <barney_cordoba@yahoo.com> wrote: --- On Fri, 4/26/13, "Clément Hermann (nodens)" <nodens2099@gmail.com> wrote: > From: "Clément Hermann (nodens)" <nodens2099@gmail.com> > Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 > To: freebsd-net@freebsd.org > Date: Friday, April 26, 2013, 7:31 AM > Hi list, > > We use pf+ALTQ for trafic shaping on some routers. > > We are switching to new servers : Dell PowerEdge R620 with 2 > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > (quad port) using igb driver. The old hardware is using em > driver, the CPU load is high but mostly due to kernel and a > large pf ruleset. > > On the new hardware, we see high CPU Interrupt load (up to > 95%), even though there is not much trafic currently (peaks > about 150Mbps and 40Kpps). All queues are used and binded to > a cpu according to top, but a lot of CPU time is spent on > igb queues (interrupt or wait). The load is fine when we > stay below 20Kpps. > > We see no mbuf shortage, no dropped packet, but there is > little margin left on CPU time (about 25% idle at best, most > of CPU time is spent on interrupts), which is disturbing. > > We have done some tuning, but to no avail : > > sysctl.conf : > > # mbufs > kern.ipc.nmbclusters=65536 > # Sockets > kern.ipc.somaxconn=8192 > net.inet.tcp.delayed_ack=0 > net.inet.tcp.sendspace=65535 > net.inet.udp.recvspace=65535 > net.inet.udp.maxdgram=57344 > net.local.stream.recvspace=65535 > net.local.stream.sendspace=65535 > # IGB > dev.igb.0.rx_processing_limit=4096 > dev.igb.1.rx_processing_limit=4096 > dev.igb.2.rx_processing_limit=4096 > dev.igb.3.rx_processing_limit=4096 > > /boot/loader.conf : > > vm.kmem_size=1G > hw.igb.max_interrupt_rate="32000" # maximum number of > interrupts/sec generated by single igb(4) (default 8000) > hw.igb.txd="2048" > > # number of transmit descriptors allocated by the > driver (2048 limit) > hw.igb.rxd="2048" > > # number of receive descriptors allocated by the > driver (2048 limit) > hw.igb.rx_process_limit="1000" # > maximum number of received packets to process at a time, The > default of 100 is > > > > > # too low for most firewalls. (-1 means > unlimited) > > Kernel HZ is 1000. > > The IGB /boot/loader.conf tuning was our last attempt, it > didn't change anything. > > Does anyone have any pointer ? How could we lower CPU > interrupt load ? should we set hw.igb.max_interrupt_rate > lower instead of higher ? > From what we saw here and there, we should be able to do > much better with this hardware. > > > relevant sysctl (igb1 and igb2 only, other interfaces are > unused) : > > sysctl dev.igb | grep -v ": 0$" > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection > version - 2.3.1 > dev.igb.1.%driver: igb > dev.igb.1.%location: slot=0 function=1 > dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 > subvendor=0x8086 subdevice=0x5001 class=0x020000 > dev.igb.1.%parent: pci5 > dev.igb.1.nvm: -1 > dev.igb.1.enable_aim: 1 > dev.igb.1.fc: 3 > dev.igb.1.rx_processing_limit: 4096 > dev.igb.1.eee_disabled: 1 > dev.igb.1.link_irq: 2 > dev.igb.1.device_control: 1209795137 > dev.igb.1.rx_control: 67141658 > dev.igb.1.interrupt_mask: 4 > dev.igb.1.extended_int_mask: 2147483981 > dev.igb.1.fc_high_water: 33168 > dev.igb.1.fc_low_water: 33152 > dev.igb.1.queue0.interrupt_rate: 71428 > dev.igb.1.queue0.txd_head: 1318 > dev.igb.1.queue0.txd_tail: 1318 > dev.igb.1.queue0.tx_packets: 84663594 > dev.igb.1.queue0.rxd_head: 717 > dev.igb.1.queue0.rxd_tail: 715 > dev.igb.1.queue0.rx_packets: 43899597 > dev.igb.1.queue0.rx_bytes: 8905556030 > dev.igb.1.queue1.interrupt_rate: 90909 > dev.igb.1.queue1.txd_head: 693 > dev.igb.1.queue1.txd_tail: 693 > dev.igb.1.queue1.tx_packets: 57543349 > dev.igb.1.queue1.rxd_head: 1033 > dev.igb.1.queue1.rxd_tail: 1032 > dev.igb.1.queue1.rx_packets: 54821897 > dev.igb.1.queue1.rx_bytes: 9944955108 > dev.igb.1.queue2.interrupt_rate: 100000 > dev.igb.1.queue2.txd_head: 350 > dev.igb.1.queue2.txd_tail: 350 > dev.igb.1.queue2.tx_packets: 62320990 > dev.igb.1.queue2.rxd_head: 1962 > dev.igb.1.queue2.rxd_tail: 1939 > dev.igb.1.queue2.rx_packets: 43909016 > dev.igb.1.queue2.rx_bytes: 8673941461 > dev.igb.1.queue3.interrupt_rate: 14925 > dev.igb.1.queue3.txd_head: 647 > dev.igb.1.queue3.txd_tail: 647 > dev.igb.1.queue3.tx_packets: 58776199 > dev.igb.1.queue3.rxd_head: 692 > dev.igb.1.queue3.rxd_tail: 691 > dev.igb.1.queue3.rx_packets: 55138996 > dev.igb.1.queue3.rx_bytes: 9310217354 > dev.igb.1.queue4.interrupt_rate: 100000 > dev.igb.1.queue4.txd_head: 1721 > dev.igb.1.queue4.txd_tail: 1721 > dev.igb.1.queue4.tx_packets: 54337209 > dev.igb.1.queue4.rxd_head: 1609 > dev.igb.1.queue4.rxd_tail: 1598 > dev.igb.1.queue4.rx_packets: 46546503 > dev.igb.1.queue4.rx_bytes: 8818182840 > dev.igb.1.queue5.interrupt_rate: 11627 > dev.igb.1.queue5.txd_head: 254 > dev.igb.1.queue5.txd_tail: 254 > dev.igb.1.queue5.tx_packets: 53117182 > dev.igb.1.queue5.rxd_head: 701 > dev.igb.1.queue5.rxd_tail: 685 > dev.igb.1.queue5.rx_packets: 43014837 > dev.igb.1.queue5.rx_bytes: 8699057447 > dev.igb.1.queue6.interrupt_rate: 55555 > dev.igb.1.queue6.txd_head: 8 > dev.igb.1.queue6.txd_tail: 8 > dev.igb.1.queue6.tx_packets: 52654088 > dev.igb.1.queue6.rxd_head: 1057 > dev.igb.1.queue6.rxd_tail: 1041 > dev.igb.1.queue6.rx_packets: 45227030 > dev.igb.1.queue6.rx_bytes: 9494489640 > dev.igb.1.queue7.interrupt_rate: 5235 > dev.igb.1.queue7.txd_head: 729 > dev.igb.1.queue7.txd_tail: 729 > dev.igb.1.queue7.tx_packets: 61926105 > dev.igb.1.queue7.rxd_head: 146 > dev.igb.1.queue7.rxd_tail: 140 > dev.igb.1.queue7.rx_packets: 51781775 > dev.igb.1.queue7.rx_bytes: 8901279226 > dev.igb.1.mac_stats.missed_packets: 1657 > dev.igb.1.mac_stats.recv_no_buff: 405 > dev.igb.1.mac_stats.total_pkts_recvd: 384332760 > dev.igb.1.mac_stats.good_pkts_recvd: 384331103 > dev.igb.1.mac_stats.bcast_pkts_recvd: 15510 > dev.igb.1.mac_stats.mcast_pkts_recvd: 52957 > dev.igb.1.mac_stats.rx_frames_64: 195496498 > dev.igb.1.mac_stats.rx_frames_65_127: 133346124 > dev.igb.1.mac_stats.rx_frames_128_255: 5254911 > dev.igb.1.mac_stats.rx_frames_256_511: 9700049 > dev.igb.1.mac_stats.rx_frames_512_1023: 16885886 > dev.igb.1.mac_stats.rx_frames_1024_1522: 23647635 > dev.igb.1.mac_stats.good_octets_recvd: 74284029276 > dev.igb.1.mac_stats.good_octets_txd: 544536708502 > dev.igb.1.mac_stats.total_pkts_txd: 485327419 > dev.igb.1.mac_stats.good_pkts_txd: 485327419 > dev.igb.1.mac_stats.bcast_pkts_txd: 72 > dev.igb.1.mac_stats.mcast_pkts_txd: 52820 > dev.igb.1.mac_stats.tx_frames_64: 57820809 > dev.igb.1.mac_stats.tx_frames_65_127: 51586341 > dev.igb.1.mac_stats.tx_frames_128_255: 7050579 > dev.igb.1.mac_stats.tx_frames_256_511: 7887126 > dev.igb.1.mac_stats.tx_frames_512_1023: 10130891 > dev.igb.1.mac_stats.tx_frames_1024_1522: 350851673 > dev.igb.1.interrupts.asserts: 551135045 > dev.igb.1.interrupts.rx_pkt_timer: 384326679 > dev.igb.1.interrupts.tx_queue_empty: 485323376 > dev.igb.1.interrupts.tx_queue_min_thresh: 6324386 > dev.igb.1.host.rx_pkt: 4424 > dev.igb.1.host.tx_good_pkt: 4043 > dev.igb.1.host.rx_good_bytes: 74284030864 > dev.igb.1.host.tx_good_bytes: 544536708502 > dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection > version - 2.3.1 > dev.igb.2.%driver: igb > dev.igb.2.%location: slot=0 function=2 > dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 > subvendor=0x8086 subdevice=0x5001 class=0x020000 > dev.igb.2.%parent: pci5 > dev.igb.2.nvm: -1 > dev.igb.2.enable_aim: 1 > dev.igb.2.fc: 3 > dev.igb.2.rx_processing_limit: 4096 > dev.igb.2.eee_disabled: 1 > dev.igb.2.link_irq: 2 > dev.igb.2.device_control: 1209795137 > dev.igb.2.rx_control: 67141658 > dev.igb.2.interrupt_mask: 4 > dev.igb.2.extended_int_mask: 2147483959 > dev.igb.2.fc_high_water: 33168 > dev.igb.2.fc_low_water: 33152 > dev.igb.2.queue0.interrupt_rate: 13698 > dev.igb.2.queue0.txd_head: 1618 > dev.igb.2.queue0.txd_tail: 1618 > dev.igb.2.queue0.tx_packets: 46401106 > dev.igb.2.queue0.rxd_head: 831 > dev.igb.2.queue0.rxd_tail: 827 > dev.igb.2.queue0.rx_packets: 69356350 > dev.igb.2.queue0.rx_bytes: 68488772907 > dev.igb.2.queue1.interrupt_rate: 5405 > dev.igb.2.queue1.txd_head: 190 > dev.igb.2.queue1.txd_tail: 190 > dev.igb.2.queue1.tx_packets: 55965886 > dev.igb.2.queue1.rxd_head: 268 > dev.igb.2.queue1.rxd_tail: 256 > dev.igb.2.queue1.rx_packets: 58958084 > dev.igb.2.queue1.rx_bytes: 69154569937 > dev.igb.2.queue2.interrupt_rate: 83333 > dev.igb.2.queue2.txd_head: 568 > dev.igb.2.queue2.txd_tail: 568 > dev.igb.2.queue2.tx_packets: 44974648 > dev.igb.2.queue2.rxd_head: 371 > dev.igb.2.queue2.rxd_tail: 219 > dev.igb.2.queue2.rx_packets: 67037407 > dev.igb.2.queue2.rx_bytes: 72042326102 > dev.igb.2.queue3.interrupt_rate: 12658 > dev.igb.2.queue3.txd_head: 867 > dev.igb.2.queue3.txd_tail: 867 > dev.igb.2.queue3.tx_packets: 55962467 > dev.igb.2.queue3.rxd_head: 85 > dev.igb.2.queue3.rxd_tail: 1953 > dev.igb.2.queue3.rx_packets: 60972965 > dev.igb.2.queue3.rx_bytes: 70397176035 > dev.igb.2.queue4.interrupt_rate: 90909 > dev.igb.2.queue4.txd_head: 1920 > dev.igb.2.queue4.txd_tail: 1920 > dev.igb.2.queue4.tx_packets: 47660931 > dev.igb.2.queue4.rxd_head: 1397 > dev.igb.2.queue4.rxd_tail: 1379 > dev.igb.2.queue4.rx_packets: 59110758 > dev.igb.2.queue4.rx_bytes: 68919201478 > dev.igb.2.queue5.interrupt_rate: 111111 > dev.igb.2.queue5.txd_head: 886 > dev.igb.2.queue5.txd_tail: 886 > dev.igb.2.queue5.tx_packets: 45103990 > dev.igb.2.queue5.rxd_head: 812 > dev.igb.2.queue5.rxd_tail: 799 > dev.igb.2.queue5.rx_packets: 59030312 > dev.igb.2.queue5.rx_bytes: 69234293962 > dev.igb.2.queue6.interrupt_rate: 5208 > dev.igb.2.queue6.txd_head: 1926 > dev.igb.2.queue6.txd_tail: 1926 > dev.igb.2.queue6.tx_packets: 46215046 > dev.igb.2.queue6.rxd_head: 692 > dev.igb.2.queue6.rxd_tail: 689 > dev.igb.2.queue6.rx_packets: 58256050 > dev.igb.2.queue6.rx_bytes: 68429172749 > dev.igb.2.queue7.interrupt_rate: 50000 > dev.igb.2.queue7.txd_head: 126 > dev.igb.2.queue7.txd_tail: 126 > dev.igb.2.queue7.tx_packets: 52451455 > dev.igb.2.queue7.rxd_head: 968 > dev.igb.2.queue7.rxd_tail: 885 > dev.igb.2.queue7.rx_packets: 65946491 > dev.igb.2.queue7.rx_bytes: 70263478849 > dev.igb.2.mac_stats.missed_packets: 958 > dev.igb.2.mac_stats.recv_no_buff: 69 > dev.igb.2.mac_stats.total_pkts_recvd: 498658079 > dev.igb.2.mac_stats.good_pkts_recvd: 498657121 > dev.igb.2.mac_stats.bcast_pkts_recvd: 16867 > dev.igb.2.mac_stats.mcast_pkts_recvd: 52957 > dev.igb.2.mac_stats.rx_frames_64: 59089332 > dev.igb.2.mac_stats.rx_frames_65_127: 52880118 > dev.igb.2.mac_stats.rx_frames_128_255: 7526966 > dev.igb.2.mac_stats.rx_frames_256_511: 8468389 > dev.igb.2.mac_stats.rx_frames_512_1023: 10434770 > dev.igb.2.mac_stats.rx_frames_1024_1522: 360257545 > dev.igb.2.mac_stats.good_octets_recvd: 558910494322 > dev.igb.2.mac_stats.good_octets_txd: 84618858153 > dev.igb.2.mac_stats.total_pkts_txd: 394726904 > dev.igb.2.mac_stats.good_pkts_txd: 394726904 > dev.igb.2.mac_stats.bcast_pkts_txd: 48 > dev.igb.2.mac_stats.mcast_pkts_txd: 52821 > dev.igb.2.mac_stats.tx_frames_64: 196605932 > dev.igb.2.mac_stats.tx_frames_65_127: 134602807 > dev.igb.2.mac_stats.tx_frames_128_255: 5705236 > dev.igb.2.mac_stats.tx_frames_256_511: 10267168 > dev.igb.2.mac_stats.tx_frames_512_1023: 17165496 > dev.igb.2.mac_stats.tx_frames_1024_1522: 30380265 > dev.igb.2.interrupts.asserts: 465994260 > dev.igb.2.interrupts.rx_pkt_timer: 498647546 > dev.igb.2.interrupts.tx_queue_empty: 394720657 > dev.igb.2.interrupts.tx_queue_min_thresh: 24424555 > dev.igb.2.host.rx_pkt: 9575 > dev.igb.2.host.tx_good_pkt: 6248 > dev.igb.2.host.rx_good_bytes: 558910513984 > dev.igb.2.host.tx_good_bytes: 84618858217 > > > Thanks for your help. > > Cheers, You're experiencing lock contention Try editing igb.c and setting the queues to 1. The multiqueue implementation in igb has a negative impact. If you have just 1 system get a dual port 82571 card and use the em driver. BC _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Sun Apr 28 22:33:01 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 01E4AB7 for <freebsd-net@freebsd.org>; Sun, 28 Apr 2013 22:33:00 +0000 (UTC) (envelope-from spinney@tilera.com) Received: from USMAMAIL.TILERA.COM (usmamail.tilera.com [12.216.194.151]) by mx1.freebsd.org (Postfix) with ESMTP id A6FDA1795 for <freebsd-net@freebsd.org>; Sun, 28 Apr 2013 22:33:00 +0000 (UTC) Received: from USMAEXCH1.tad.internal.tilera.com ([fe80::709c:7a3e:ae82:7a6e]) by USMAExch2.tad.internal.tilera.com ([fe80::408c:3921:ab63:6a87%11]) with mapi; Sun, 28 Apr 2013 18:31:51 -0400 From: Barry Spinney <spinney@tilera.com> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: TF_NEEDSYN flag never set. Thread-Topic: TF_NEEDSYN flag never set. Thread-Index: Ac5EXUkf2910zLpuSFy6dwGLhSsWmQ=Date: Sun, 28 Apr 2013 22:31:48 +0000 Message-ID: <73BC01C897E9F642AC962BF311A45ACB100B0057@USMAExch1.tad.internal.tilera.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Barry Spinney <spinney@tilera.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>; List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sun, 28 Apr 2013 22:33:01 -0000 I am sorry if this is a dumb question, but I was trying to understand the FreeBSD TCP stack, and In particular I was trying to understand the use of the TF_NEEDSYN flag. This flag is referenced a number of times in tcp_input.c and tcp_output.c, but I don't think that it can ever be set. In particular grepping through the "../src/sys/netinet", one discovers that the only code that can set this flag is lines 1018 and 1020 of tcp_input.c. But, it appears to me that none of the lines in tcp_input.c from 999 thru 1023 are even reachable! The reason they are not reachable is because just ahead of them are the following lines: if (!syncache_add(&inc, &to, th, &so, m)) goto drop; if (so == NULL) { ... // uninteresting lines, but no gotos return; } ... /unreachable code here Studying syncache_add (in file tcp_syncache.c), reveals three return statements. One of the returns, returns the value 0, which will cause the "goto drop" to be executed. The other two returns, return both the value 1 AND set "*sop = NULL", which should cause the following "if (so == NULL)" to execute the subsequent return statement. Is this intentional? (i.e. dead code awaiting future development?), or a bug? Or I am going blind to something obvious? Thanx Barry Spinney. (p.s. I doubt it matters which version of code, but to be precise this is from the /pub/FreeBSD/development/tarballs named "src_stable_6.tar.gz" dated "4/21/2013 01:15 AM", gotten from ftp1.us.freebsd.org<ftp://ftp1.us.freebsd.org>)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1367175573.97340.YahooMailClassic>