Date: Mon, 20 Jul 2020 15:05:50 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Konstantin Belousov <kostikbel@gmail.com> Cc: net@freebsd.org Subject: Re: mlx5 interrupts Message-ID: <20200720120550.GJ2015@zxy.spb.ru> In-Reply-To: <20200720113148.GL44314@kib.kiev.ua> References: <20200719161207.GA2033@zxy.spb.ru> <20200720104339.GK44314@kib.kiev.ua> <20200720110426.GI2015@zxy.spb.ru> <20200720113148.GL44314@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jul 20, 2020 at 02:31:48PM +0300, Konstantin Belousov wrote: > On Mon, Jul 20, 2020 at 02:04:26PM +0300, Slawa Olhovchenkov wrote: > > On Mon, Jul 20, 2020 at 01:43:39PM +0300, Konstantin Belousov wrote: > > > > > On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote: > > > > Can anybody explain what purpose of unnamed interrupts of Mellanox > > > > ConnectX-5 cards? > > > > > > > > I am see 19 interrupts per card. I am mean last 16 is RX queue. > > > It is more correct to describe them as completion queue interrupts, they > > > serve both rx and tx workqueues. > > > > > > > What about first 3? > > > One is for firmware memory requests notifications for UMA. > > > Two others are firmware command completion and async events (like PHY > > > module events or hw state notifications). > > > > > > > Also I am see very high rate for irq287/irq306 -- is this good? > > > You did not demostrated what you mean by 'very high rate'. > > > 20 req/sec does not count as 'high', and this is the highest I see > > > in your message. > > > > Currently this is idle server and totaly interrupts count for firmware > > command completion is 50x more compared to completion queue > > interrupts. I am afraid to flooded at prodation load. > Did you reflashed firmware to the latest supported release ? > If not, do that. Don't release number in logs, only show by melanox utilites? > Why are you sure that this is command completion and not events ? You words: One is for firmware memory requests notifications for UMA firmware command completion async events -- I mean you named in order. Unfortunately all interrupts named identicaly > Set sysctl dev.mce.0.conf.debug_stats=1 then look at the output > of sysctl dev.mce.0.hw_ctx_debug. > dev.mce.0.hw_ctx_debug: pages irq 286 command irq 287 async irq 288 channel 0 rq 353 cq 1025 irq 289 channel 0 tc 0 sq 352 cq 1024 irq 289 > > Also, completion queue interrupts may be pined to dedicaded CPU, what > > about command completion? > Command completion is normal MSI-X interrupt, but I think that the official > recommendation is to not fiddle with it. Commands are not used for normal > tx/rx path, they only provide management and statistics. > > > > > > > > > > > # vmstat -i | grep -e ^int -e mlx > > > > interrupt total rate > > > > irq286: mlx5_core0 1 0 > > > > irq287: mlx5_core0 5135992 20 > > > > irq288: mlx5_core0 1 0 > > > > irq289: mlx5_core0 76408 0 > > > > irq290: mlx5_core0 43054 0 > > > > irq291: mlx5_core0 93826 0 > > > > irq292: mlx5_core0 39457 0 > > > > irq293: mlx5_core0 36141 0 > > > > irq294: mlx5_core0 65526 0 > > > > irq295: mlx5_core0 53399 0 > > > > irq296: mlx5_core0 120885 0 > > > > irq297: mlx5_core0 140690 1 > > > > irq298: mlx5_core0 193578 1 > > > > irq299: mlx5_core0 178332 1 > > > > irq300: mlx5_core0 75334 0 > > > > irq301: mlx5_core0 207118 1 > > > > irq302: mlx5_core0 108803 0 > > > > irq303: mlx5_core0 24356 0 > > > > irq304: mlx5_core0 26713 0 > > > > irq305: mlx5_core1 1 0 > > > > irq306: mlx5_core1 5136296 20 > > > > irq307: mlx5_core1 1 0 > > > > irq308: mlx5_core1 3634544 14 > > > > irq309: mlx5_core1 22860 0 > > > > irq310: mlx5_core1 564441 2 > > > > irq311: mlx5_core1 30503 0 > > > > irq312: mlx5_core1 115549 0 > > > > irq313: mlx5_core1 49815 0 > > > > irq314: mlx5_core1 10272 0 > > > > irq315: mlx5_core1 85875 0 > > > > irq316: mlx5_core1 134251 1 > > > > irq317: mlx5_core1 25151 0 > > > > irq318: mlx5_core1 73376 0 > > > > irq319: mlx5_core1 5879 0 > > > > irq320: mlx5_core1 39515 0 > > > > irq321: mlx5_core1 5390 0 > > > > irq322: mlx5_core1 22726 0 > > > > irq323: mlx5_core1 60408 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200720120550.GJ2015>