Date: Thu, 13 Feb 2020 22:49:13 -0800 From: Rudy <crapsh@monkeybrains.net> To: net@FreeBSD.org Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 Message-ID: <a4c98e33-3aae-f08b-4132-52350a33a56c@monkeybrains.net> In-Reply-To: <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net> References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net> <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/12/20 7:21 PM, Rudy wrote: > > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load > (and high interrupts). Update, I have found the chelsio_affinity script and run that, doesn't seem to help. I did notice the T6 card is on numa-domain 0 and the T5 cards on on numa-domain 1. Does that matter? Here is the relevant part of dmesg: t6nex0: <Chelsio T62100-LP-CR> mem 0xc5300000-0xc537ffff,0xc4000000-0xc4ffffff,0xc5984000-0xc5985fff irq 48 at device 0.4 numa-domain 0 on pci12 cc0: <port 0> numa-domain 0 on t6nex0 cc1: <port 1> numa-domain 0 on t6nex0 t6nex0: PCIe gen3 x16, 2 ports, 34 MSI-X interrupts, 66 eq, 33 iq t5nex0: <Chelsio T540-LP-CR> mem 0xed300000-0xed37ffff,0xec000000-0xecffffff,0xed984000-0xed985fff irq 88 at device 0.4 numa-domain 1 on pci18 cxl0: <port 0> numa-domain 1 on t5nex0 cxl1: <port 1> numa-domain 1 on t5nex0 cxl2: <port 2> numa-domain 1 on t5nex0 cxl3: <port 3> numa-domain 1 on t5nex0 t5nex0: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq t5nex1: <Chelsio T540-LP-CR> mem 0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xfb984000-0xfb985fff irq 98 at device 0.4 numa-domain 1 on pci20 cxl4: <port 0> numa-domain 1 on t5nex1 cxl5: <port 1> numa-domain 1 on t5nex1 cxl6: <port 2> numa-domain 1 on t5nex1 cxl7: <port 3> numa-domain 1 on t5nex1 t5nex1: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq Box does have 20 cores and hyper-threading is disabled. sysctl hw.ncpu hw.ncpu: 20 I've added updated /boot/loader.conf to the following: hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 machdep.hyperthreading_allowed="0" Box works fine unless I turn on TX on the cc1 device (I am filtering routes learned from that interface via BGP). Rudy
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a4c98e33-3aae-f08b-4132-52350a33a56c>