From owner-freebsd-net@freebsd.org Mon Jul 20 10:43:47 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DB7B0358EBD for ; Mon, 20 Jul 2020 10:43:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4B9JHR4zmXz3XnK for ; Mon, 20 Jul 2020 10:43:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id A9478358BE7; Mon, 20 Jul 2020 10:43:47 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A90D3358B6E for ; Mon, 20 Jul 2020 10:43:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B9JHQ6tsQz3XsV for ; Mon, 20 Jul 2020 10:43:46 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id 06KAhdmg039989 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 20 Jul 2020 13:43:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 06KAhdmg039989 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id 06KAhdej039988; Mon, 20 Jul 2020 13:43:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 20 Jul 2020 13:43:39 +0300 From: Konstantin Belousov To: Slawa Olhovchenkov Cc: net@freebsd.org Subject: Re: mlx5 interrupts Message-ID: <20200720104339.GK44314@kib.kiev.ua> References: <20200719161207.GA2033@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200719161207.GA2033@zxy.spb.ru> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4B9JHQ6tsQz3XsV X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [1.23 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.14)[-0.136]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_SPAM_SHORT(0.85)[0.852]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.52)[0.517]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jul 2020 10:43:47 -0000 On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote: > Can anybody explain what purpose of unnamed interrupts of Mellanox > ConnectX-5 cards? > > I am see 19 interrupts per card. I am mean last 16 is RX queue. It is more correct to describe them as completion queue interrupts, they serve both rx and tx workqueues. > What about first 3? One is for firmware memory requests notifications for UMA. Two others are firmware command completion and async events (like PHY module events or hw state notifications). > Also I am see very high rate for irq287/irq306 -- is this good? You did not demostrated what you mean by 'very high rate'. 20 req/sec does not count as 'high', and this is the highest I see in your message. > > # vmstat -i | grep -e ^int -e mlx > interrupt total rate > irq286: mlx5_core0 1 0 > irq287: mlx5_core0 5135992 20 > irq288: mlx5_core0 1 0 > irq289: mlx5_core0 76408 0 > irq290: mlx5_core0 43054 0 > irq291: mlx5_core0 93826 0 > irq292: mlx5_core0 39457 0 > irq293: mlx5_core0 36141 0 > irq294: mlx5_core0 65526 0 > irq295: mlx5_core0 53399 0 > irq296: mlx5_core0 120885 0 > irq297: mlx5_core0 140690 1 > irq298: mlx5_core0 193578 1 > irq299: mlx5_core0 178332 1 > irq300: mlx5_core0 75334 0 > irq301: mlx5_core0 207118 1 > irq302: mlx5_core0 108803 0 > irq303: mlx5_core0 24356 0 > irq304: mlx5_core0 26713 0 > irq305: mlx5_core1 1 0 > irq306: mlx5_core1 5136296 20 > irq307: mlx5_core1 1 0 > irq308: mlx5_core1 3634544 14 > irq309: mlx5_core1 22860 0 > irq310: mlx5_core1 564441 2 > irq311: mlx5_core1 30503 0 > irq312: mlx5_core1 115549 0 > irq313: mlx5_core1 49815 0 > irq314: mlx5_core1 10272 0 > irq315: mlx5_core1 85875 0 > irq316: mlx5_core1 134251 1 > irq317: mlx5_core1 25151 0 > irq318: mlx5_core1 73376 0 > irq319: mlx5_core1 5879 0 > irq320: mlx5_core1 39515 0 > irq321: mlx5_core1 5390 0 > irq322: mlx5_core1 22726 0 > irq323: mlx5_core1 60408 0