From owner-freebsd-net@FreeBSD.ORG Sun Apr 28 17:07:06 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 782B9266 for ; Sun, 28 Apr 2013 17:07:06 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vb0-x22f.google.com (mail-vb0-x22f.google.com [IPv6:2607:f8b0:400c:c02::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 390581D76 for ; Sun, 28 Apr 2013 17:07:06 +0000 (UTC) Received: by mail-vb0-f47.google.com with SMTP id x14so107289vbb.34 for ; Sun, 28 Apr 2013 10:07:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=JfwfK+AWm0zI9FM21uPJ0iVl9M+jJtLTrnZT459rbys=; b=G1HU1rpOgCzmTtilVHgGxHXwuokSR9eavN+8vsH038NY/c9kCnL2ogaL5pRXX4O+7m Mg298WT3xBoFmm8uoPG0XW5/hDxKLxEkngNhx+4XDTIDA7Y4Vc/mvq4SOv1+Z3lQRQOm DQ6ZxrkzZbW1r/rReG3kK8fFl4L8rP8Ccvll3eG1vC5nbRZdfoY2/xYnGCdkoO6JZr9/ qSIibao2wfb/RiAbkoWA73i1B3DywFkV/08FmRzFZfWvTKGOmZokld/lwhgsLKys1GgP CGmy2WwAYxoEdbLPCyhciq8EiOnozrl1M9Vmeyjk0ci8lrpchGpr95yzlZOG202QhAwt mcow== MIME-Version: 1.0 X-Received: by 10.52.22.202 with SMTP id g10mr26913858vdf.126.1367168825511; Sun, 28 Apr 2013 10:07:05 -0700 (PDT) Received: by 10.220.55.143 with HTTP; Sun, 28 Apr 2013 10:07:05 -0700 (PDT) In-Reply-To: <1367158902.96158.YahooMailClassic@web121603.mail.ne1.yahoo.com> References: <517A657B.7060003@gmail.com> <1367158902.96158.YahooMailClassic@web121603.mail.ne1.yahoo.com> Date: Sun, 28 Apr 2013 10:07:05 -0700 Message-ID: Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 From: Jack Vogel To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Net , =?ISO-8859-1?Q?Cl=E9ment_Hermann_=28nodens=29?= X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Apr 2013 17:07:06 -0000 Try setting your queues to 1, run some tests, then try setting your queues to 2, then to 4... its called tuning, and rather than just pontificating about it, which Barney so loves to do, you can discover what works best. I ran tests last week preparing for a new driver version and found the best results came not only while tweaking queues, but also ring size, and I could see changes based on the buf ring size.... There are lots of things that may improve or degrade performance depending on the workload. Jack On Sun, Apr 28, 2013 at 7:21 AM, Barney Cordoba w= rote: > > > --- On Fri, 4/26/13, "Cl=E9ment Hermann (nodens)" > wrote: > > > From: "Cl=E9ment Hermann (nodens)" > > Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 > > To: freebsd-net@freebsd.org > > Date: Friday, April 26, 2013, 7:31 AM > > Hi list, > > > > We use pf+ALTQ for trafic shaping on some routers. > > > > We are switching to new servers : Dell PowerEdge R620 with 2 > > 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 > > (quad port) using igb driver. The old hardware is using em > > driver, the CPU load is high but mostly due to kernel and a > > large pf ruleset. > > > > On the new hardware, we see high CPU Interrupt load (up to > > 95%), even though there is not much trafic currently (peaks > > about 150Mbps and 40Kpps). All queues are used and binded to > > a cpu according to top, but a lot of CPU time is spent on > > igb queues (interrupt or wait). The load is fine when we > > stay below 20Kpps. > > > > We see no mbuf shortage, no dropped packet, but there is > > little margin left on CPU time (about 25% idle at best, most > > of CPU time is spent on interrupts), which is disturbing. > > > > We have done some tuning, but to no avail : > > > > sysctl.conf : > > > > # mbufs > > kern.ipc.nmbclusters=3D65536 > > # Sockets > > kern.ipc.somaxconn=3D8192 > > net.inet.tcp.delayed_ack=3D0 > > net.inet.tcp.sendspace=3D65535 > > net.inet.udp.recvspace=3D65535 > > net.inet.udp.maxdgram=3D57344 > > net.local.stream.recvspace=3D65535 > > net.local.stream.sendspace=3D65535 > > # IGB > > dev.igb.0.rx_processing_limit=3D4096 > > dev.igb.1.rx_processing_limit=3D4096 > > dev.igb.2.rx_processing_limit=3D4096 > > dev.igb.3.rx_processing_limit=3D4096 > > > > /boot/loader.conf : > > > > vm.kmem_size=3D1G > > hw.igb.max_interrupt_rate=3D"32000" # maximum number of > > interrupts/sec generated by single igb(4) (default 8000) > > hw.igb.txd=3D"2048" > > > > # number of transmit descriptors allocated by the > > driver (2048 limit) > > hw.igb.rxd=3D"2048" > > > > # number of receive descriptors allocated by the > > driver (2048 limit) > > hw.igb.rx_process_limit=3D"1000" # > > maximum number of received packets to process at a time, The > > default of 100 is > > > > > > > > > > # too low for most firewalls. (-1 means > > unlimited) > > > > Kernel HZ is 1000. > > > > The IGB /boot/loader.conf tuning was our last attempt, it > > didn't change anything. > > > > Does anyone have any pointer ? How could we lower CPU > > interrupt load ? should we set hw.igb.max_interrupt_rate > > lower instead of higher ? > > From what we saw here and there, we should be able to do > > much better with this hardware. > > > > > > relevant sysctl (igb1 and igb2 only, other interfaces are > > unused) : > > > > sysctl dev.igb | grep -v ": 0$" > > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection > > version - 2.3.1 > > dev.igb.1.%driver: igb > > dev.igb.1.%location: slot=3D0 function=3D1 > > dev.igb.1.%pnpinfo: vendor=3D0x8086 device=3D0x1521 > > subvendor=3D0x8086 subdevice=3D0x5001 class=3D0x020000 > > dev.igb.1.%parent: pci5 > > dev.igb.1.nvm: -1 > > dev.igb.1.enable_aim: 1 > > dev.igb.1.fc: 3 > > dev.igb.1.rx_processing_limit: 4096 > > dev.igb.1.eee_disabled: 1 > > dev.igb.1.link_irq: 2 > > dev.igb.1.device_control: 1209795137 > > dev.igb.1.rx_control: 67141658 > > dev.igb.1.interrupt_mask: 4 > > dev.igb.1.extended_int_mask: 2147483981 > > dev.igb.1.fc_high_water: 33168 > > dev.igb.1.fc_low_water: 33152 > > dev.igb.1.queue0.interrupt_rate: 71428 > > dev.igb.1.queue0.txd_head: 1318 > > dev.igb.1.queue0.txd_tail: 1318 > > dev.igb.1.queue0.tx_packets: 84663594 > > dev.igb.1.queue0.rxd_head: 717 > > dev.igb.1.queue0.rxd_tail: 715 > > dev.igb.1.queue0.rx_packets: 43899597 > > dev.igb.1.queue0.rx_bytes: 8905556030 > > dev.igb.1.queue1.interrupt_rate: 90909 > > dev.igb.1.queue1.txd_head: 693 > > dev.igb.1.queue1.txd_tail: 693 > > dev.igb.1.queue1.tx_packets: 57543349 > > dev.igb.1.queue1.rxd_head: 1033 > > dev.igb.1.queue1.rxd_tail: 1032 > > dev.igb.1.queue1.rx_packets: 54821897 > > dev.igb.1.queue1.rx_bytes: 9944955108 > > dev.igb.1.queue2.interrupt_rate: 100000 > > dev.igb.1.queue2.txd_head: 350 > > dev.igb.1.queue2.txd_tail: 350 > > dev.igb.1.queue2.tx_packets: 62320990 > > dev.igb.1.queue2.rxd_head: 1962 > > dev.igb.1.queue2.rxd_tail: 1939 > > dev.igb.1.queue2.rx_packets: 43909016 > > dev.igb.1.queue2.rx_bytes: 8673941461 > > dev.igb.1.queue3.interrupt_rate: 14925 > > dev.igb.1.queue3.txd_head: 647 > > dev.igb.1.queue3.txd_tail: 647 > > dev.igb.1.queue3.tx_packets: 58776199 > > dev.igb.1.queue3.rxd_head: 692 > > dev.igb.1.queue3.rxd_tail: 691 > > dev.igb.1.queue3.rx_packets: 55138996 > > dev.igb.1.queue3.rx_bytes: 9310217354 > > dev.igb.1.queue4.interrupt_rate: 100000 > > dev.igb.1.queue4.txd_head: 1721 > > dev.igb.1.queue4.txd_tail: 1721 > > dev.igb.1.queue4.tx_packets: 54337209 > > dev.igb.1.queue4.rxd_head: 1609 > > dev.igb.1.queue4.rxd_tail: 1598 > > dev.igb.1.queue4.rx_packets: 46546503 > > dev.igb.1.queue4.rx_bytes: 8818182840 > > dev.igb.1.queue5.interrupt_rate: 11627 > > dev.igb.1.queue5.txd_head: 254 > > dev.igb.1.queue5.txd_tail: 254 > > dev.igb.1.queue5.tx_packets: 53117182 > > dev.igb.1.queue5.rxd_head: 701 > > dev.igb.1.queue5.rxd_tail: 685 > > dev.igb.1.queue5.rx_packets: 43014837 > > dev.igb.1.queue5.rx_bytes: 8699057447 > > dev.igb.1.queue6.interrupt_rate: 55555 > > dev.igb.1.queue6.txd_head: 8 > > dev.igb.1.queue6.txd_tail: 8 > > dev.igb.1.queue6.tx_packets: 52654088 > > dev.igb.1.queue6.rxd_head: 1057 > > dev.igb.1.queue6.rxd_tail: 1041 > > dev.igb.1.queue6.rx_packets: 45227030 > > dev.igb.1.queue6.rx_bytes: 9494489640 > > dev.igb.1.queue7.interrupt_rate: 5235 > > dev.igb.1.queue7.txd_head: 729 > > dev.igb.1.queue7.txd_tail: 729 > > dev.igb.1.queue7.tx_packets: 61926105 > > dev.igb.1.queue7.rxd_head: 146 > > dev.igb.1.queue7.rxd_tail: 140 > > dev.igb.1.queue7.rx_packets: 51781775 > > dev.igb.1.queue7.rx_bytes: 8901279226 > > dev.igb.1.mac_stats.missed_packets: 1657 > > dev.igb.1.mac_stats.recv_no_buff: 405 > > dev.igb.1.mac_stats.total_pkts_recvd: 384332760 > > dev.igb.1.mac_stats.good_pkts_recvd: 384331103 > > dev.igb.1.mac_stats.bcast_pkts_recvd: 15510 > > dev.igb.1.mac_stats.mcast_pkts_recvd: 52957 > > dev.igb.1.mac_stats.rx_frames_64: 195496498 > > dev.igb.1.mac_stats.rx_frames_65_127: 133346124 > > dev.igb.1.mac_stats.rx_frames_128_255: 5254911 > > dev.igb.1.mac_stats.rx_frames_256_511: 9700049 > > dev.igb.1.mac_stats.rx_frames_512_1023: 16885886 > > dev.igb.1.mac_stats.rx_frames_1024_1522: 23647635 > > dev.igb.1.mac_stats.good_octets_recvd: 74284029276 > > dev.igb.1.mac_stats.good_octets_txd: 544536708502 > > dev.igb.1.mac_stats.total_pkts_txd: 485327419 > > dev.igb.1.mac_stats.good_pkts_txd: 485327419 > > dev.igb.1.mac_stats.bcast_pkts_txd: 72 > > dev.igb.1.mac_stats.mcast_pkts_txd: 52820 > > dev.igb.1.mac_stats.tx_frames_64: 57820809 > > dev.igb.1.mac_stats.tx_frames_65_127: 51586341 > > dev.igb.1.mac_stats.tx_frames_128_255: 7050579 > > dev.igb.1.mac_stats.tx_frames_256_511: 7887126 > > dev.igb.1.mac_stats.tx_frames_512_1023: 10130891 > > dev.igb.1.mac_stats.tx_frames_1024_1522: 350851673 > > dev.igb.1.interrupts.asserts: 551135045 > > dev.igb.1.interrupts.rx_pkt_timer: 384326679 > > dev.igb.1.interrupts.tx_queue_empty: 485323376 > > dev.igb.1.interrupts.tx_queue_min_thresh: 6324386 > > dev.igb.1.host.rx_pkt: 4424 > > dev.igb.1.host.tx_good_pkt: 4043 > > dev.igb.1.host.rx_good_bytes: 74284030864 > > dev.igb.1.host.tx_good_bytes: 544536708502 > > dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection > > version - 2.3.1 > > dev.igb.2.%driver: igb > > dev.igb.2.%location: slot=3D0 function=3D2 > > dev.igb.2.%pnpinfo: vendor=3D0x8086 device=3D0x1521 > > subvendor=3D0x8086 subdevice=3D0x5001 class=3D0x020000 > > dev.igb.2.%parent: pci5 > > dev.igb.2.nvm: -1 > > dev.igb.2.enable_aim: 1 > > dev.igb.2.fc: 3 > > dev.igb.2.rx_processing_limit: 4096 > > dev.igb.2.eee_disabled: 1 > > dev.igb.2.link_irq: 2 > > dev.igb.2.device_control: 1209795137 > > dev.igb.2.rx_control: 67141658 > > dev.igb.2.interrupt_mask: 4 > > dev.igb.2.extended_int_mask: 2147483959 > > dev.igb.2.fc_high_water: 33168 > > dev.igb.2.fc_low_water: 33152 > > dev.igb.2.queue0.interrupt_rate: 13698 > > dev.igb.2.queue0.txd_head: 1618 > > dev.igb.2.queue0.txd_tail: 1618 > > dev.igb.2.queue0.tx_packets: 46401106 > > dev.igb.2.queue0.rxd_head: 831 > > dev.igb.2.queue0.rxd_tail: 827 > > dev.igb.2.queue0.rx_packets: 69356350 > > dev.igb.2.queue0.rx_bytes: 68488772907 > > dev.igb.2.queue1.interrupt_rate: 5405 > > dev.igb.2.queue1.txd_head: 190 > > dev.igb.2.queue1.txd_tail: 190 > > dev.igb.2.queue1.tx_packets: 55965886 > > dev.igb.2.queue1.rxd_head: 268 > > dev.igb.2.queue1.rxd_tail: 256 > > dev.igb.2.queue1.rx_packets: 58958084 > > dev.igb.2.queue1.rx_bytes: 69154569937 > > dev.igb.2.queue2.interrupt_rate: 83333 > > dev.igb.2.queue2.txd_head: 568 > > dev.igb.2.queue2.txd_tail: 568 > > dev.igb.2.queue2.tx_packets: 44974648 > > dev.igb.2.queue2.rxd_head: 371 > > dev.igb.2.queue2.rxd_tail: 219 > > dev.igb.2.queue2.rx_packets: 67037407 > > dev.igb.2.queue2.rx_bytes: 72042326102 > > dev.igb.2.queue3.interrupt_rate: 12658 > > dev.igb.2.queue3.txd_head: 867 > > dev.igb.2.queue3.txd_tail: 867 > > dev.igb.2.queue3.tx_packets: 55962467 > > dev.igb.2.queue3.rxd_head: 85 > > dev.igb.2.queue3.rxd_tail: 1953 > > dev.igb.2.queue3.rx_packets: 60972965 > > dev.igb.2.queue3.rx_bytes: 70397176035 > > dev.igb.2.queue4.interrupt_rate: 90909 > > dev.igb.2.queue4.txd_head: 1920 > > dev.igb.2.queue4.txd_tail: 1920 > > dev.igb.2.queue4.tx_packets: 47660931 > > dev.igb.2.queue4.rxd_head: 1397 > > dev.igb.2.queue4.rxd_tail: 1379 > > dev.igb.2.queue4.rx_packets: 59110758 > > dev.igb.2.queue4.rx_bytes: 68919201478 > > dev.igb.2.queue5.interrupt_rate: 111111 > > dev.igb.2.queue5.txd_head: 886 > > dev.igb.2.queue5.txd_tail: 886 > > dev.igb.2.queue5.tx_packets: 45103990 > > dev.igb.2.queue5.rxd_head: 812 > > dev.igb.2.queue5.rxd_tail: 799 > > dev.igb.2.queue5.rx_packets: 59030312 > > dev.igb.2.queue5.rx_bytes: 69234293962 > > dev.igb.2.queue6.interrupt_rate: 5208 > > dev.igb.2.queue6.txd_head: 1926 > > dev.igb.2.queue6.txd_tail: 1926 > > dev.igb.2.queue6.tx_packets: 46215046 > > dev.igb.2.queue6.rxd_head: 692 > > dev.igb.2.queue6.rxd_tail: 689 > > dev.igb.2.queue6.rx_packets: 58256050 > > dev.igb.2.queue6.rx_bytes: 68429172749 > > dev.igb.2.queue7.interrupt_rate: 50000 > > dev.igb.2.queue7.txd_head: 126 > > dev.igb.2.queue7.txd_tail: 126 > > dev.igb.2.queue7.tx_packets: 52451455 > > dev.igb.2.queue7.rxd_head: 968 > > dev.igb.2.queue7.rxd_tail: 885 > > dev.igb.2.queue7.rx_packets: 65946491 > > dev.igb.2.queue7.rx_bytes: 70263478849 > > dev.igb.2.mac_stats.missed_packets: 958 > > dev.igb.2.mac_stats.recv_no_buff: 69 > > dev.igb.2.mac_stats.total_pkts_recvd: 498658079 > > dev.igb.2.mac_stats.good_pkts_recvd: 498657121 > > dev.igb.2.mac_stats.bcast_pkts_recvd: 16867 > > dev.igb.2.mac_stats.mcast_pkts_recvd: 52957 > > dev.igb.2.mac_stats.rx_frames_64: 59089332 > > dev.igb.2.mac_stats.rx_frames_65_127: 52880118 > > dev.igb.2.mac_stats.rx_frames_128_255: 7526966 > > dev.igb.2.mac_stats.rx_frames_256_511: 8468389 > > dev.igb.2.mac_stats.rx_frames_512_1023: 10434770 > > dev.igb.2.mac_stats.rx_frames_1024_1522: 360257545 > > dev.igb.2.mac_stats.good_octets_recvd: 558910494322 > > dev.igb.2.mac_stats.good_octets_txd: 84618858153 > > dev.igb.2.mac_stats.total_pkts_txd: 394726904 > > dev.igb.2.mac_stats.good_pkts_txd: 394726904 > > dev.igb.2.mac_stats.bcast_pkts_txd: 48 > > dev.igb.2.mac_stats.mcast_pkts_txd: 52821 > > dev.igb.2.mac_stats.tx_frames_64: 196605932 > > dev.igb.2.mac_stats.tx_frames_65_127: 134602807 > > dev.igb.2.mac_stats.tx_frames_128_255: 5705236 > > dev.igb.2.mac_stats.tx_frames_256_511: 10267168 > > dev.igb.2.mac_stats.tx_frames_512_1023: 17165496 > > dev.igb.2.mac_stats.tx_frames_1024_1522: 30380265 > > dev.igb.2.interrupts.asserts: 465994260 > > dev.igb.2.interrupts.rx_pkt_timer: 498647546 > > dev.igb.2.interrupts.tx_queue_empty: 394720657 > > dev.igb.2.interrupts.tx_queue_min_thresh: 24424555 > > dev.igb.2.host.rx_pkt: 9575 > > dev.igb.2.host.tx_good_pkt: 6248 > > dev.igb.2.host.rx_good_bytes: 558910513984 > > dev.igb.2.host.tx_good_bytes: 84618858217 > > > > > > Thanks for your help. > > > > Cheers, > > You're experiencing lock contention > > Try editing igb.c and setting the queues to 1. The multiqueue > implementation in igb has a negative impact. > > If you have just 1 system get a dual port 82571 card and use the > em driver. > > BC > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >