From owner-freebsd-net@freebsd.org Mon Jul 20 12:05:55 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0604335C8B6 for ; Mon, 20 Jul 2020 12:05:55 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4B9L6B4dmhz3dJc for ; Mon, 20 Jul 2020 12:05:54 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.nyi.freebsd.org (Postfix) id 9F1B735C97A; Mon, 20 Jul 2020 12:05:54 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9EE3435C7ED for ; Mon, 20 Jul 2020 12:05:54 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9L695n10z3dBj for ; Mon, 20 Jul 2020 12:05:53 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1jxUYI-000PZq-Fp; Mon, 20 Jul 2020 15:05:50 +0300 Date: Mon, 20 Jul 2020 15:05:50 +0300 From: Slawa Olhovchenkov To: Konstantin Belousov Cc: net@freebsd.org Subject: Re: mlx5 interrupts Message-ID: <20200720120550.GJ2015@zxy.spb.ru> References: <20200719161207.GA2033@zxy.spb.ru> <20200720104339.GK44314@kib.kiev.ua> <20200720110426.GI2015@zxy.spb.ru> <20200720113148.GL44314@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200720113148.GL44314@kib.kiev.ua> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 4B9L695n10z3dBj X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [1.32 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.10)[0.101]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.18)[-0.182]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.50)[0.502]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 12:05:55 -0000 On Mon, Jul 20, 2020 at 02:31:48PM +0300, Konstantin Belousov wrote: > On Mon, Jul 20, 2020 at 02:04:26PM +0300, Slawa Olhovchenkov wrote: > > On Mon, Jul 20, 2020 at 01:43:39PM +0300, Konstantin Belousov wrote: > > > > > On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote: > > > > Can anybody explain what purpose of unnamed interrupts of Mellanox > > > > ConnectX-5 cards? > > > > > > > > I am see 19 interrupts per card. I am mean last 16 is RX queue. > > > It is more correct to describe them as completion queue interrupts, they > > > serve both rx and tx workqueues. > > > > > > > What about first 3? > > > One is for firmware memory requests notifications for UMA. > > > Two others are firmware command completion and async events (like PHY > > > module events or hw state notifications). > > > > > > > Also I am see very high rate for irq287/irq306 -- is this good? > > > You did not demostrated what you mean by 'very high rate'. > > > 20 req/sec does not count as 'high', and this is the highest I see > > > in your message. > > > > Currently this is idle server and totaly interrupts count for firmware > > command completion is 50x more compared to completion queue > > interrupts. I am afraid to flooded at prodation load. > Did you reflashed firmware to the latest supported release ? > If not, do that. Don't release number in logs, only show by melanox utilites? > Why are you sure that this is command completion and not events ? You words: One is for firmware memory requests notifications for UMA firmware command completion async events -- I mean you named in order. Unfortunately all interrupts named identicaly > Set sysctl dev.mce.0.conf.debug_stats=1 then look at the output > of sysctl dev.mce.0.hw_ctx_debug. > dev.mce.0.hw_ctx_debug: pages irq 286 command irq 287 async irq 288 channel 0 rq 353 cq 1025 irq 289 channel 0 tc 0 sq 352 cq 1024 irq 289 > > Also, completion queue interrupts may be pined to dedicaded CPU, what > > about command completion? > Command completion is normal MSI-X interrupt, but I think that the official > recommendation is to not fiddle with it. Commands are not used for normal > tx/rx path, they only provide management and statistics. > > > > > > > > > > > # vmstat -i | grep -e ^int -e mlx > > > > interrupt total rate > > > > irq286: mlx5_core0 1 0 > > > > irq287: mlx5_core0 5135992 20 > > > > irq288: mlx5_core0 1 0 > > > > irq289: mlx5_core0 76408 0 > > > > irq290: mlx5_core0 43054 0 > > > > irq291: mlx5_core0 93826 0 > > > > irq292: mlx5_core0 39457 0 > > > > irq293: mlx5_core0 36141 0 > > > > irq294: mlx5_core0 65526 0 > > > > irq295: mlx5_core0 53399 0 > > > > irq296: mlx5_core0 120885 0 > > > > irq297: mlx5_core0 140690 1 > > > > irq298: mlx5_core0 193578 1 > > > > irq299: mlx5_core0 178332 1 > > > > irq300: mlx5_core0 75334 0 > > > > irq301: mlx5_core0 207118 1 > > > > irq302: mlx5_core0 108803 0 > > > > irq303: mlx5_core0 24356 0 > > > > irq304: mlx5_core0 26713 0 > > > > irq305: mlx5_core1 1 0 > > > > irq306: mlx5_core1 5136296 20 > > > > irq307: mlx5_core1 1 0 > > > > irq308: mlx5_core1 3634544 14 > > > > irq309: mlx5_core1 22860 0 > > > > irq310: mlx5_core1 564441 2 > > > > irq311: mlx5_core1 30503 0 > > > > irq312: mlx5_core1 115549 0 > > > > irq313: mlx5_core1 49815 0 > > > > irq314: mlx5_core1 10272 0 > > > > irq315: mlx5_core1 85875 0 > > > > irq316: mlx5_core1 134251 1 > > > > irq317: mlx5_core1 25151 0 > > > > irq318: mlx5_core1 73376 0 > > > > irq319: mlx5_core1 5879 0 > > > > irq320: mlx5_core1 39515 0 > > > > irq321: mlx5_core1 5390 0 > > > > irq322: mlx5_core1 22726 0 > > > > irq323: mlx5_core1 60408 0