Date: Fri, 14 Feb 2020 19:00:11 +0100 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@freebsd.org> To: Rudy <crapsh@monkeybrains.net> Cc: freebsd-net@freebsd.org Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 Message-ID: <CA%2Bq%2BTcrUyzfNtLGD4Vtc3a0v5MHcxVvED=eK57aGL21LTQzL4w@mail.gmail.com> In-Reply-To: <428f3cdf-9035-90a7-14f8-f294c2131682@monkeybrains.net> References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net> <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net> <a4c98e33-3aae-f08b-4132-52350a33a56c@monkeybrains.net> <428f3cdf-9035-90a7-14f8-f294c2131682@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Feb 14, 2020 at 6:25 PM Rudy <crapsh@monkeybrains.net> wrote: > On 2/12/20 7:21 PM, Rudy wrote: > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load (and > high interrupts). > > > Looking better! I made some changes based on BSDRP which I hadn't known > about -- I think ifqmaxlen was the tunable I overlooked. > > # > > https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local > net.link.ifqmaxlen="16384" > > This net.link.ifqmaxlen was set to help in case of lagg usage: I was not aware it could improve your use case. >From your first post, it looks like your setup is a 2 packages, 10 cores, 20 threads (disabled). And you have configured your Chelsio to use 16 queues (hw.cxgbe.Xrx=16): It's a good think to have a power of 2 number of queues with Chelsio, but I'm not sure it's a good idea to spread those queue across the 2 packages. So perhaps you should try: 1. To reduce queues to 8 queues and bind them to the local domain 2. Or keeping 16 queues, but re-enabling HyperThreading and bing them to the local domain too. (on -head with recent CPU and machdep.hyperthreading_intr_allowed, using hyper-threading improve forwarding performance). But anyway even with 16 queues spread over 2 domains, you should have better performance: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650v4_2x12Cores-Chelsio_T520-CR/hw.cxgbe.nXxq/results/fbsd12-stable.r354440.BSDRP.1.96/README.md Notice that I never monitoring the CPU load during my benches. Increasing the hw.cxgbe.holdoff_timer_idx was a good idea: I would expect lower interrupt usage too. Did you monitor the QPI link usage ? (kldload cpuctl && pcm-numa.x)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcrUyzfNtLGD4Vtc3a0v5MHcxVvED=eK57aGL21LTQzL4w>