Date: Tue, 25 Feb 2020 18:57:36 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: freebsd-net@freebsd.org Subject: Re: Chelsio NETMAP performance Message-ID: <20200225155736.GN8012@zxy.spb.ru> In-Reply-To: <20200206015914.GA7071@x270> References: <20200203201728.GC8028@zxy.spb.ru> <863de9e1-42cc-6f3a-5c1f-1bf737714c9f@FreeBSD.org> <20200203222321.GB8012@zxy.spb.ru> <6868f207-d054-3d45-b60d-eaf7115760c1@FreeBSD.org> <20200204162005.GC8012@zxy.spb.ru> <3a8dfebd-aa26-84ad-a03a-0271b61a89a3@FreeBSD.org> <20200205113832.GE8012@zxy.spb.ru> <20200206015914.GA7071@x270>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 05, 2020 at 05:59:14PM -0800, Navdeep Parhar wrote: > On Wed, Feb 05, 2020 at 02:38:32PM +0300, Slawa Olhovchenkov wrote: > > On Tue, Feb 04, 2020 at 12:37:08PM -0800, Navdeep Parhar wrote: > > > > > >> nm_holdoff_tmr_idx is a 0-based index into the list above. So if the > > > >> tmr idx is 0 you are using the 0th (first) value from the list of > > > >> timers. Try increasing nm_holdoff_tmr_idx and see if that brings down > > > >> the interrupt rate under control. > > > >> > > > >> # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3/4/5 > > > > > > > > OK, interrupt rate go down, but interrupt time about same. > > > > (interrupt rate for intel card about 0, compared to 25% chelsio). > > > > > > I think iflib runs a lot of stuff in taskqueues rather than the driver > > > ithread so the CPU accounting may vary. Use dtrace to see if > > > > Don't think this is impact: worker's CPU core w/o any syscalls and > > only w/ bunding workker thread and NIC irq handler show about 100% > > user CPU time. > > > > May be some cache-miss work performed later, at poll(2) time in case > > of intel driver compared to chelsio (do at interrupt time)? > > Could be. While we are here, is it possible for you to try the patches > in these two? > > https://reviews.freebsd.org/D17868 about ~10% performance drop > https://reviews.freebsd.org/D17869 about ~3-5% performance rise. > > > > > netmap_rx_irq is being called by an ithread or a taskqueue to figure out > > > what driver does what. > > > > Can you explain some more? > > I am not sure about dtrace probe to use and later evaluation > > # dtrace -n 'fbt::netmap_rx_irq:entry {stack()}' > > Take a look at the stack and see if it's an ithread or one of iflib's > taskqueues that called netmap_rx_irq. > > Regards, > Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200225155736.GN8012>