Date: Wed, 5 Feb 2020 17:59:14 -0800 From: Navdeep Parhar <np@FreeBSD.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NETMAP performance Message-ID: <20200206015914.GA7071@x270> In-Reply-To: <20200205113832.GE8012@zxy.spb.ru> References: <20200203201728.GC8028@zxy.spb.ru> <863de9e1-42cc-6f3a-5c1f-1bf737714c9f@FreeBSD.org> <20200203222321.GB8012@zxy.spb.ru> <6868f207-d054-3d45-b60d-eaf7115760c1@FreeBSD.org> <20200204162005.GC8012@zxy.spb.ru> <3a8dfebd-aa26-84ad-a03a-0271b61a89a3@FreeBSD.org> <20200205113832.GE8012@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 05, 2020 at 02:38:32PM +0300, Slawa Olhovchenkov wrote: > On Tue, Feb 04, 2020 at 12:37:08PM -0800, Navdeep Parhar wrote: > > > >> nm_holdoff_tmr_idx is a 0-based index into the list above. So if the > > >> tmr idx is 0 you are using the 0th (first) value from the list of > > >> timers. Try increasing nm_holdoff_tmr_idx and see if that brings down > > >> the interrupt rate under control. > > >> > > >> # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3/4/5 > > > > > > OK, interrupt rate go down, but interrupt time about same. > > > (interrupt rate for intel card about 0, compared to 25% chelsio). > > > > I think iflib runs a lot of stuff in taskqueues rather than the driver > > ithread so the CPU accounting may vary. Use dtrace to see if > > Don't think this is impact: worker's CPU core w/o any syscalls and > only w/ bunding workker thread and NIC irq handler show about 100% > user CPU time. > > May be some cache-miss work performed later, at poll(2) time in case > of intel driver compared to chelsio (do at interrupt time)? Could be. While we are here, is it possible for you to try the patches in these two? https://reviews.freebsd.org/D17868 https://reviews.freebsd.org/D17869 > > > netmap_rx_irq is being called by an ithread or a taskqueue to figure out > > what driver does what. > > Can you explain some more? > I am not sure about dtrace probe to use and later evaluation # dtrace -n 'fbt::netmap_rx_irq:entry {stack()}' Take a look at the stack and see if it's an ithread or one of iflib's taskqueues that called netmap_rx_irq. Regards, Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200206015914.GA7071>