Date: Mon, 5 Mar 2007 14:41:52 -0500 (EST) From: Andrew Gallatin <gallatin@cs.duke.edu> To: Robert Watson <rwatson@FreeBSD.org> Cc: freebsd-net@FreeBSD.org, freebsd-current@FreeBSD.org, Andre Oppermann <andre@FreeBSD.org>, kmacy@FreeBSD.org, jhb@FreeBSD.org Subject: Re: New optimized soreceive_stream() for TCP sockets, proof of concept Message-ID: <17900.29312.198932.502783@grasshopper.cs.duke.edu> In-Reply-To: <20070305182755.S31701@fledge.watson.org> References: <45E8276D.60105@freebsd.org> <17900.24574.751134.397740@grasshopper.cs.duke.edu> <20070305182755.S31701@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson writes: > On Mon, 5 Mar 2007, Andrew Gallatin wrote: > > > With the patch, we finally seem to be performance competative on the receive > > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of > > those OSes do much better (saturate the link with jumbos) when CPU affinity > > is used to bind the interrupt handler and netserver process to different > > cores on the same socket. I imagine FreeBSD may be able to do even better > > if it ever grows CPU affinity support for both interrupt handlers and > > processes. With the patch, it performs at least as well, if not better > > than, Solaris and Linux do without CPU affinity. > > I don't have numbers in front of me, and am currently packing for a trip to > Tokyo so won't find them before traveling, but my experience has been that > binding the ithread to a specific CPU is very helpful in improving receive > performance. You can slap a sched_bind(0) into the interrupt handler the > first time it runs and it should stick appropriately, and add a sysctl to > sched_bind() for a user process as a hack to test it out. OK, So I did a hack which binds anything which calls accept() to CPU1, and then hacked the intr handler of my driver to bind it to CPU0. I saw no improvement. Darn. BTW, doing binding like this seems to entirely eliminate the out-of-order packets I see when net.isr.direct=0. Drew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?17900.29312.198932.502783>