From owner-freebsd-net@freebsd.org Wed Feb 8 09:56:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5AC6CD5234 for ; Wed, 8 Feb 2017 09:56:49 +0000 (UTC) (envelope-from farrokhi@FreeBSD.org) Received: from mail.farrokhi.net (mail.farrokhi.net [79.127.49.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5366C1B40 for ; Wed, 8 Feb 2017 09:56:49 +0000 (UTC) (envelope-from farrokhi@FreeBSD.org) Received: from [192.168.0.107] (unknown [79.127.49.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: freebsd@farrokhi.net) by mail.farrokhi.net (Postfix) with ESMTPSA id 2D6F545571 for ; Wed, 8 Feb 2017 13:26:37 +0330 (IRST) From: "Babak Farrokhi" To: "FreeBSD Net" Subject: ixgbe interrupt rate on 11-STABLE Date: Wed, 08 Feb 2017 13:26:36 +0330 Message-ID: <08C3F17E-53F6-4CDC-BFEA-D2B294A8C704@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_85FDD487-A78F-4AB7-96FB-0272D5D69606_="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Mailer: MailMate (1.9.6r5342) X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2017 09:56:49 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_85FDD487-A78F-4AB7-96FB-0272D5D69606_= Content-Type: text/plain; markup=markdown Content-Transfer-Encoding: quoted-printable Hi all, I encountered a new behavior with ixgbe interrupt rates on 11-STABLE whic= h = was not present in 10-STABLE. = We disable aim and manually set the interrupt rate. On 10-STABLE the inte= rrupt = rate will not exceed the configured value, but it is not the case in 11-S= TABLE. I am wondering if this is expected behavior or not. Here is what I see: # sysctl dev.ix | grep interrupt_rate dev.ix.1.queue7.interrupt_rate: 500000 dev.ix.1.queue6.interrupt_rate: 500000 dev.ix.1.queue5.interrupt_rate: 100000 dev.ix.1.queue4.interrupt_rate: 500000 dev.ix.1.queue3.interrupt_rate: 500000 dev.ix.1.queue2.interrupt_rate: 500000 dev.ix.1.queue1.interrupt_rate: 500000 dev.ix.1.queue0.interrupt_rate: 500000 dev.ix.0.queue7.interrupt_rate: 500000 dev.ix.0.queue6.interrupt_rate: 500000 dev.ix.0.queue5.interrupt_rate: 100000 dev.ix.0.queue4.interrupt_rate: 5813 dev.ix.0.queue3.interrupt_rate: 500000 dev.ix.0.queue2.interrupt_rate: 500000 dev.ix.0.queue1.interrupt_rate: 500000 dev.ix.0.queue0.interrupt_rate: 500000 Our setup: # sysctl hw.ix hw.ix.rxd: 4096 hw.ix.txd: 4096 hw.ix.num_queues: 8 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 0 hw.ix.tx_process_limit: -1 hw.ix.rx_process_limit: -1 hw.ix.max_interrupt_rate: 16000 hw.ix.enable_aim: 0 # sysctl dev.ix | grep aim dev.ix.1.enable_aim: 0 dev.ix.0.enable_aim: 0 # vmstat -ai | grep ix irq265: ix0:q0 1252479695 10058 irq266: ix0:q1 1232959515 9901 irq267: ix0:q2 1227101328 9854 irq268: ix0:q3 1229457979 9873 irq269: ix0:q4 1237070699 9934 irq270: ix0:q5 1231407684 9889 irq271: ix0:q6 1234032662 9910 irq272: ix0:q7 1230403197 9881 irq273: ix0:link 107 0 irq274: ix1:q0 2928503268 23517 irq275: ix1:q1 2526264197 20287 irq276: ix1:q2 2543700719 20427 irq277: ix1:q3 2552022026 20494 irq278: ix1:q4 2584441166 20754 irq279: ix1:q5 2563593862 20587 irq280: ix1:q6 2567670402 20619 irq281: ix1:q7 2581949876 20734 irq282: ix1:link 2 0 Hardware: # pciconf -lv | grep -A3 ix ix0@pci0:5:0:0: class=3D0x020000 card=3D0x7b118086 chip=3D0x154d8086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet 10G 2P X520 Adapter' class =3D network -- ix1@pci0:5:0:1: class=3D0x020000 card=3D0x7b118086 chip=3D0x154d8086 rev=3D= 0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D 'Ethernet 10G 2P X520 Adapter' class =3D network # dmesg | grep ^ix ix0: = port 0x2020-0x203f mem 0x92300000-0x923fffff,0x92404000-0x92407fff irq 34= at device 0.0 numa-domain 0 on pci5 ix0: Using MSIX interrupts with 9 vectors ix0: ixgbe_allocate_msix: number of queues (8) !=3D number of RSS buckets= (128); performance will be impacted. ix0: Ethernet address: a0:36:9f:da:05:4c ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: Error 2 setting up SR-IOV ix0: netmap queues/slots: TX 8/4096, RX 8/4096 ix1: = port 0x2000-0x201f mem 0x92200000-0x922fffff,0x92400000-0x92403fff irq 37= at device 0.1 numa-domain 0 on pci5 ix1: Using MSIX interrupts with 9 vectors ix1: ixgbe_allocate_msix: number of queues (8) !=3D number of RSS buckets= (128); performance will be impacted. ix1: Ethernet address: a0:36:9f:da:05:4e ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: Error 2 setting up SR-IOV ix1: netmap queues/slots: TX 8/4096, RX 8/4096 Any ideas? Thanks, Babak --=_MailMate_85FDD487-A78F-4AB7-96FB-0272D5D69606_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJ8BAEBCABmBQJYmutUXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGMDgxNUY4ODYxQkYyREVBRjI2MUU5QzE2 QjI2N0FEODVENjMyRTlBAAoJEGsmethdYy6aQ2UP/iFS5b2ylOGHzYV4oe741bRB aQREDtHcFPOlNq8T1P9jA+ijEZpBFwEA3GQdPC5+JKvqBNrCeJFwgp3Ejbu94UKw /C9TUxZuCqbSTu599X4FvsmeN+YO0WD4kxrqlZ5ynR6gKGCXH+e283JmlRZH2/59 s8tZEVUAizgif8YOW4LGj6+3OR+oNxh9bCOyu9eNH8kc66BUekp2w/1A0/7TaEew 4kr2BV3+PL1hnAbYVc7uj6gHiT4RCBafk6Y+rNqBpPKtmzrQGV53ghWaZ1/+1Mkd nXVThlPQ3JfuZO9Gg1csznslIJca3bXG+4RJLIYQiy7+7KbmOM2k38nzqoUGS0i5 DXsdnkaopi1MsSo0CE3QXQLrMhVgUl07jIgCD6iHQvEYa0TDmsXEkxtfICCk6YBZ Iy2oT+U+Mj/J/mBM5hKJ9dwPKI43O3voq/bQRTtxiJLitQ6XHMIX+AJwUhOIvb5U dAsDuSWAQn8gmf9FrB1nWLc0dJR8pc9jCAEJdt9jT48TLs8Weu3Dvh/7ErzuSwd1 prl497/EoCRsrons6iIh5+sZciRa5eB8dJsKO6upi41NwBQ8+9/4daBhpv14WKT6 1Eb4cBX2kSDHikJ8a4NGky4g/sbVDeOFp5eTO5ciC4HtB975Eae5iwjSuX9uJKnK IVOs+2YjHoHf0mxkSuQK =4oTR -----END PGP SIGNATURE----- --=_MailMate_85FDD487-A78F-4AB7-96FB-0272D5D69606_=--