Date: Wed, 08 Feb 2017 13:26:36 +0330 From: "Babak Farrokhi" <farrokhi@FreeBSD.org> To: "FreeBSD Net" <freebsd-net@freebsd.org> Subject: ixgbe interrupt rate on 11-STABLE Message-ID: <08C3F17E-53F6-4CDC-BFEA-D2B294A8C704@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hi all,
I encountered a new behavior with ixgbe interrupt rates on 11-STABLE which
was not present in 10-STABLE.
We disable aim and manually set the interrupt rate. On 10-STABLE the interrupt
rate will not exceed the configured value, but it is not the case in 11-STABLE.
I am wondering if this is expected behavior or not.
Here is what I see:
# sysctl dev.ix | grep interrupt_rate
dev.ix.1.queue7.interrupt_rate: 500000
dev.ix.1.queue6.interrupt_rate: 500000
dev.ix.1.queue5.interrupt_rate: 100000
dev.ix.1.queue4.interrupt_rate: 500000
dev.ix.1.queue3.interrupt_rate: 500000
dev.ix.1.queue2.interrupt_rate: 500000
dev.ix.1.queue1.interrupt_rate: 500000
dev.ix.1.queue0.interrupt_rate: 500000
dev.ix.0.queue7.interrupt_rate: 500000
dev.ix.0.queue6.interrupt_rate: 500000
dev.ix.0.queue5.interrupt_rate: 100000
dev.ix.0.queue4.interrupt_rate: 5813
dev.ix.0.queue3.interrupt_rate: 500000
dev.ix.0.queue2.interrupt_rate: 500000
dev.ix.0.queue1.interrupt_rate: 500000
dev.ix.0.queue0.interrupt_rate: 500000
Our setup:
# sysctl hw.ix
hw.ix.rxd: 4096
hw.ix.txd: 4096
hw.ix.num_queues: 8
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 0
hw.ix.tx_process_limit: -1
hw.ix.rx_process_limit: -1
hw.ix.max_interrupt_rate: 16000
hw.ix.enable_aim: 0
# sysctl dev.ix | grep aim
dev.ix.1.enable_aim: 0
dev.ix.0.enable_aim: 0
# vmstat -ai | grep ix
irq265: ix0:q0 1252479695 10058
irq266: ix0:q1 1232959515 9901
irq267: ix0:q2 1227101328 9854
irq268: ix0:q3 1229457979 9873
irq269: ix0:q4 1237070699 9934
irq270: ix0:q5 1231407684 9889
irq271: ix0:q6 1234032662 9910
irq272: ix0:q7 1230403197 9881
irq273: ix0:link 107 0
irq274: ix1:q0 2928503268 23517
irq275: ix1:q1 2526264197 20287
irq276: ix1:q2 2543700719 20427
irq277: ix1:q3 2552022026 20494
irq278: ix1:q4 2584441166 20754
irq279: ix1:q5 2563593862 20587
irq280: ix1:q6 2567670402 20619
irq281: ix1:q7 2581949876 20734
irq282: ix1:link 2 0
Hardware:
# pciconf -lv | grep -A3 ix
ix0@pci0:5:0:0: class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet 10G 2P X520 Adapter'
class = network
--
ix1@pci0:5:0:1: class=0x020000 card=0x7b118086 chip=0x154d8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet 10G 2P X520 Adapter'
class = network
# dmesg | grep ^ix
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> port 0x2020-0x203f mem 0x92300000-0x923fffff,0x92404000-0x92407fff irq 34 at device 0.0 numa-domain 0 on pci5
ix0: Using MSIX interrupts with 9 vectors
ix0: ixgbe_allocate_msix: number of queues (8) != number of RSS buckets (128); performance will be impacted.
ix0: Ethernet address: a0:36:9f:da:05:4c
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: Error 2 setting up SR-IOV
ix0: netmap queues/slots: TX 8/4096, RX 8/4096
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> port 0x2000-0x201f mem 0x92200000-0x922fffff,0x92400000-0x92403fff irq 37 at device 0.1 numa-domain 0 on pci5
ix1: Using MSIX interrupts with 9 vectors
ix1: ixgbe_allocate_msix: number of queues (8) != number of RSS buckets (128); performance will be impacted.
ix1: Ethernet address: a0:36:9f:da:05:4e
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: Error 2 setting up SR-IOV
ix1: netmap queues/slots: TX 8/4096, RX 8/4096
Any ideas?
Thanks,
Babak
[-- Attachment #2 --]
-----BEGIN PGP SIGNATURE-----
iQJ8BAEBCABmBQJYmutUXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGMDgxNUY4ODYxQkYyREVBRjI2MUU5QzE2
QjI2N0FEODVENjMyRTlBAAoJEGsmethdYy6aQ2UP/iFS5b2ylOGHzYV4oe741bRB
aQREDtHcFPOlNq8T1P9jA+ijEZpBFwEA3GQdPC5+JKvqBNrCeJFwgp3Ejbu94UKw
/C9TUxZuCqbSTu599X4FvsmeN+YO0WD4kxrqlZ5ynR6gKGCXH+e283JmlRZH2/59
s8tZEVUAizgif8YOW4LGj6+3OR+oNxh9bCOyu9eNH8kc66BUekp2w/1A0/7TaEew
4kr2BV3+PL1hnAbYVc7uj6gHiT4RCBafk6Y+rNqBpPKtmzrQGV53ghWaZ1/+1Mkd
nXVThlPQ3JfuZO9Gg1csznslIJca3bXG+4RJLIYQiy7+7KbmOM2k38nzqoUGS0i5
DXsdnkaopi1MsSo0CE3QXQLrMhVgUl07jIgCD6iHQvEYa0TDmsXEkxtfICCk6YBZ
Iy2oT+U+Mj/J/mBM5hKJ9dwPKI43O3voq/bQRTtxiJLitQ6XHMIX+AJwUhOIvb5U
dAsDuSWAQn8gmf9FrB1nWLc0dJR8pc9jCAEJdt9jT48TLs8Weu3Dvh/7ErzuSwd1
prl497/EoCRsrons6iIh5+sZciRa5eB8dJsKO6upi41NwBQ8+9/4daBhpv14WKT6
1Eb4cBX2kSDHikJ8a4NGky4g/sbVDeOFp5eTO5ciC4HtB975Eae5iwjSuX9uJKnK
IVOs+2YjHoHf0mxkSuQK
=4oTR
-----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?08C3F17E-53F6-4CDC-BFEA-D2B294A8C704>