Date: Wed, 23 Nov 2022 14:40:31 -0500 From: mike tancsa <mike@sentex.net> To: Navdeep Parhar <nparhar@gmail.com>, Freebsd performance <freebsd-performance@freebsd.org> Subject: Re: Chelsio Forwarding performance and RELENG_13 vs RELENG_12 (solved) Message-ID: <f33caf13-dc2c-96c0-d3c9-2d17e0e344bc@sentex.net> In-Reply-To: <63424978-a10f-a88b-2b3e-eb80d0f29f51@sentex.net> References: <7b86e3fe-62e4-7b3e-f4bf-30e4894db9db@sentex.net> <92cdf4b8-2209-ec44-8151-a59b9e8f1504@gmail.com> <b741dd17-f6eb-66c3-82f6-52b87475704f@sentex.net> <fa98a400-75b4-6453-0b40-ae098ac3ab59@gmail.com> <ceff6a91-929d-a3f9-e99e-03952b125ae2@sentex.net> <8166abfe-a796-2cf0-ade2-de08df8eecd2@gmail.com> <39ca9375-e742-618e-5020-dda5fa24ac0a@sentex.net> <63424978-a10f-a88b-2b3e-eb80d0f29f51@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/3/2022 2:20 PM, mike tancsa wrote: > Yes, I think 4 queues are enough for 10G. >>> >> Sadly, no luck. Still about the same rate of overflows :( >> >> > FYI, I worked around the issue by using two 520-CR NICs instead of the > one 540-CR NIC and performance is solid again with no dropped packets > > Another configuration point on this. Moving to RELENG_13 has some different defaults with respect to power/performance ratios for my motherboard and CPU (SuperMicro X11SCH-F, Xeon(R) E-2226G). On RELENG_13, the hwpstate_intel attaches by default and is used to scale up and down the CPU frequency. I am guessing due to the somewhat bursty nature of the load, the CPU scaling down to 800Hz could not scale back up fast enough to deal with a sudden burst of traffic going from say 300Mb/s to 800Mb/s and some packets would overflow the NIC's buffers. Printing out the cpu frequency once per second, it would be constantly floating up and down from 900 to 4300. At first, I couldnt quite get my head around the fact that the most lost packets would happen at the lowest pps periods. Once I started to graph the cpu freq, CPU temp, pps, Mb/s, the pattern really stood out. Sure enough, setting dev.hwpstate_intel.0.epp=0 on the cores from the default of 50 (see HWPSTATE_INTEL(4) ) made the difference. # sysctl -a dev.cpufreq.0.freq_driver dev.cpufreq.0.freq_driver: hwpstate_intel0 # ---Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f33caf13-dc2c-96c0-d3c9-2d17e0e344bc>