From owner-freebsd-net@freebsd.org Mon Jul 20 11:04:31 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CFEC235A2A1 for ; Mon, 20 Jul 2020 11:04:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4B9JlM3LWnz3Ykn for ; Mon, 20 Jul 2020 11:04:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.nyi.freebsd.org (Postfix) id 72A88359FBE; Mon, 20 Jul 2020 11:04:31 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 726AC35A041 for ; Mon, 20 Jul 2020 11:04:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9JlK6M62z3Z3h for ; Mon, 20 Jul 2020 11:04:29 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1jxTas-000P2G-KD; Mon, 20 Jul 2020 14:04:26 +0300 Date: Mon, 20 Jul 2020 14:04:26 +0300 From: Slawa Olhovchenkov To: Konstantin Belousov Cc: net@freebsd.org Subject: Re: mlx5 interrupts Message-ID: <20200720110426.GI2015@zxy.spb.ru> References: <20200719161207.GA2033@zxy.spb.ru> <20200720104339.GK44314@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200720104339.GK44314@kib.kiev.ua> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 4B9JlK6M62z3Z3h X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [1.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.10)[0.102]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.00)[-0.003]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.50)[0.501]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 11:04:31 -0000 On Mon, Jul 20, 2020 at 01:43:39PM +0300, Konstantin Belousov wrote: > On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote: > > Can anybody explain what purpose of unnamed interrupts of Mellanox > > ConnectX-5 cards? > > > > I am see 19 interrupts per card. I am mean last 16 is RX queue. > It is more correct to describe them as completion queue interrupts, they > serve both rx and tx workqueues. > > > What about first 3? > One is for firmware memory requests notifications for UMA. > Two others are firmware command completion and async events (like PHY > module events or hw state notifications). > > > Also I am see very high rate for irq287/irq306 -- is this good? > You did not demostrated what you mean by 'very high rate'. > 20 req/sec does not count as 'high', and this is the highest I see > in your message. Currently this is idle server and totaly interrupts count for firmware command completion is 50x more compared to completion queue interrupts. I am afraid to flooded at prodation load. Also, completion queue interrupts may be pined to dedicaded CPU, what about command completion? > > > > # vmstat -i | grep -e ^int -e mlx > > interrupt total rate > > irq286: mlx5_core0 1 0 > > irq287: mlx5_core0 5135992 20 > > irq288: mlx5_core0 1 0 > > irq289: mlx5_core0 76408 0 > > irq290: mlx5_core0 43054 0 > > irq291: mlx5_core0 93826 0 > > irq292: mlx5_core0 39457 0 > > irq293: mlx5_core0 36141 0 > > irq294: mlx5_core0 65526 0 > > irq295: mlx5_core0 53399 0 > > irq296: mlx5_core0 120885 0 > > irq297: mlx5_core0 140690 1 > > irq298: mlx5_core0 193578 1 > > irq299: mlx5_core0 178332 1 > > irq300: mlx5_core0 75334 0 > > irq301: mlx5_core0 207118 1 > > irq302: mlx5_core0 108803 0 > > irq303: mlx5_core0 24356 0 > > irq304: mlx5_core0 26713 0 > > irq305: mlx5_core1 1 0 > > irq306: mlx5_core1 5136296 20 > > irq307: mlx5_core1 1 0 > > irq308: mlx5_core1 3634544 14 > > irq309: mlx5_core1 22860 0 > > irq310: mlx5_core1 564441 2 > > irq311: mlx5_core1 30503 0 > > irq312: mlx5_core1 115549 0 > > irq313: mlx5_core1 49815 0 > > irq314: mlx5_core1 10272 0 > > irq315: mlx5_core1 85875 0 > > irq316: mlx5_core1 134251 1 > > irq317: mlx5_core1 25151 0 > > irq318: mlx5_core1 73376 0 > > irq319: mlx5_core1 5879 0 > > irq320: mlx5_core1 39515 0 > > irq321: mlx5_core1 5390 0 > > irq322: mlx5_core1 22726 0 > > irq323: mlx5_core1 60408 0