Date: Thu, 1 Oct 2020 10:10:42 +0200 From: Hans Petter Selasky <hps@selasky.org> To: =?UTF-8?Q?Michal_Van=c4=8do?= <michal@microwave.sk>, freebsd-net@freebsd.org Subject: Re: mlx5 irq Message-ID: <94978a05-94c6-cc55-229c-5a3c5352b29a@selasky.org> In-Reply-To: <0aa09fcc-dfcc-005e-8834-2a758ba6a03f@microwave.sk> References: <0aa09fcc-dfcc-005e-8834-2a758ba6a03f@microwave.sk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-10-01 09:39, Michal Vančo via freebsd-net wrote: > Hi Hi Michal, > I have a server with one Mellanox ConnectX-4 adapter and the following > CPU configuration (SMT disabled): > > # dmesg | grep SMP > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads > FreeBSD/SMP Online: 2 package(s) x 8 core(s) > > What I don't understand is the number of IRQs allocated for each mlx5_core: > > # vmstat -i | grep mlx5_core > irq320: mlx5_core0 1 0 > irq321: mlx5_core0 18646775 84 > irq322: mlx5_core0 21 0 > irq323: mlx5_core0 97793 0 > irq324: mlx5_core0 84685 0 > irq325: mlx5_core0 89288 0 > irq326: mlx5_core0 93564 0 > irq327: mlx5_core0 86892 0 > irq328: mlx5_core0 99141 0 > irq329: mlx5_core0 86695 0 > irq330: mlx5_core0 104023 0 > irq331: mlx5_core0 85238 0 > irq332: mlx5_core0 88387 0 > irq333: mlx5_core0 93310221 420 ^^^ it appears you have some application which is using a single TCP connection heavily. Then the traffic doesn't get distributed. > irq334: mlx5_core0 1135906 5 > irq335: mlx5_core0 85394 0 > irq336: mlx5_core0 88361 0 > irq337: mlx5_core0 88826 0 > irq338: mlx5_core0 17909515 81 > irq339: mlx5_core1 1 0 > irq340: mlx5_core1 18646948 84 > irq341: mlx5_core1 25 0 > irq342: mlx5_core1 208684 1 > irq343: mlx5_core1 91567 0 > irq344: mlx5_core1 88340 0 > irq345: mlx5_core1 92597 0 > irq346: mlx5_core1 85108 0 > irq347: mlx5_core1 98858 0 > irq348: mlx5_core1 88103 0 > irq349: mlx5_core1 104906 0 > irq350: mlx5_core1 84947 0 > irq351: mlx5_core1 99767 0 > irq352: mlx5_core1 9482571 43 > irq353: mlx5_core1 1724267 8 > irq354: mlx5_core1 96698 0 > irq355: mlx5_core1 473324 2 > irq356: mlx5_core1 86760 0 > irq357: mlx5_core1 11590861 52 > > I expected number of IRQs to be equal number of CPUS. According to > Mellanox docs, I should be able to pin each interrupt to specific core > to loadbalance. How can I do this in this case when number of IRQs is > larger than number of cores? Is there any way to lower the number of > interrupts? > You can lower the number of interrupts by changing the coalescing sysctl's in the mce.<N>.conf tree. dev.mce.0.conf.tx_coalesce_pkts: 32 dev.mce.0.conf.tx_coalesce_usecs: 16 dev.mce.0.conf.rx_coalesce_pkts: 32 dev.mce.0.conf.rx_coalesce_usecs: 3 For example 1024 pkts and 125 us. And also set the queue size bigger than 1024 pkts: dev.mce.0.conf.rx_queue_size: 1024 dev.mce.0.conf.tx_queue_size: 1024 --HPS
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?94978a05-94c6-cc55-229c-5a3c5352b29a>