From owner-freebsd-net@freebsd.org Tue Feb 25 15:57:42 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2B74925EF7C for ; Tue, 25 Feb 2020 15:57:42 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48Rk9010gpz4NB7 for ; Tue, 25 Feb 2020 15:57:39 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1j6caW-000CIV-Vx for freebsd-net@freebsd.org; Tue, 25 Feb 2020 18:57:36 +0300 Date: Tue, 25 Feb 2020 18:57:36 +0300 From: Slawa Olhovchenkov To: freebsd-net@freebsd.org Subject: Re: Chelsio NETMAP performance Message-ID: <20200225155736.GN8012@zxy.spb.ru> References: <20200203201728.GC8028@zxy.spb.ru> <863de9e1-42cc-6f3a-5c1f-1bf737714c9f@FreeBSD.org> <20200203222321.GB8012@zxy.spb.ru> <6868f207-d054-3d45-b60d-eaf7115760c1@FreeBSD.org> <20200204162005.GC8012@zxy.spb.ru> <3a8dfebd-aa26-84ad-a03a-0271b61a89a3@FreeBSD.org> <20200205113832.GE8012@zxy.spb.ru> <20200206015914.GA7071@x270> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200206015914.GA7071@x270> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 48Rk9010gpz4NB7 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [0.40 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.84)[-0.837,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.26)[0.258,0]; RCVD_TLS_LAST(0.00)[]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.08)[asn: 5495(0.38), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2020 15:57:42 -0000 On Wed, Feb 05, 2020 at 05:59:14PM -0800, Navdeep Parhar wrote: > On Wed, Feb 05, 2020 at 02:38:32PM +0300, Slawa Olhovchenkov wrote: > > On Tue, Feb 04, 2020 at 12:37:08PM -0800, Navdeep Parhar wrote: > > > > > >> nm_holdoff_tmr_idx is a 0-based index into the list above. So if the > > > >> tmr idx is 0 you are using the 0th (first) value from the list of > > > >> timers. Try increasing nm_holdoff_tmr_idx and see if that brings down > > > >> the interrupt rate under control. > > > >> > > > >> # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3/4/5 > > > > > > > > OK, interrupt rate go down, but interrupt time about same. > > > > (interrupt rate for intel card about 0, compared to 25% chelsio). > > > > > > I think iflib runs a lot of stuff in taskqueues rather than the driver > > > ithread so the CPU accounting may vary. Use dtrace to see if > > > > Don't think this is impact: worker's CPU core w/o any syscalls and > > only w/ bunding workker thread and NIC irq handler show about 100% > > user CPU time. > > > > May be some cache-miss work performed later, at poll(2) time in case > > of intel driver compared to chelsio (do at interrupt time)? > > Could be. While we are here, is it possible for you to try the patches > in these two? > > https://reviews.freebsd.org/D17868 about ~10% performance drop > https://reviews.freebsd.org/D17869 about ~3-5% performance rise. > > > > > netmap_rx_irq is being called by an ithread or a taskqueue to figure out > > > what driver does what. > > > > Can you explain some more? > > I am not sure about dtrace probe to use and later evaluation > > # dtrace -n 'fbt::netmap_rx_irq:entry {stack()}' > > Take a look at the stack and see if it's an ithread or one of iflib's > taskqueues that called netmap_rx_irq. > > Regards, > Navdeep