Date: Fri, 4 Feb 2005 22:48:04 +0200 From: Ruslan Ermilov <ru@FreeBSD.org> To: Guy Helmer <ghelmer@palisadesys.com> Cc: freebsd-net@FreeBSD.org Subject: Re: Netgraph performance question Message-ID: <20050204204804.GC71363@ip.net.ua> In-Reply-To: <4203AAE3.4090906@palisadesys.com> References: <4203AAE3.4090906@palisadesys.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi Guy, On Fri, Feb 04, 2005 at 11:03:31AM -0600, Guy Helmer wrote: > A while back, Maxim Konovalov made a commit to usr.sbin/ngctl/main.c to > increase its socket receive buffer size to help 'ngctl list' deal with a > big number of nodes, and Ruslan Ermilov responded that setting sysctls > net.graph.recvspace=200000 and net.graph.maxdgram=200000 was a good idea > on a system with a large number of nodes. > > I'm getting what I consider to be sub-par performance under FreeBSD 5.3 > from a userland program using ngsockets connected into ng_tee to play > with packets that are traversing a ng_bridge, and I finally have an > opportunity to look into this. I say "sub-par" because when we've > tested this configuration using three 2.8GHz Xeon machines with Gigabit > Ethernet interfaces at 1000Mbps full-duplex, we obtained peak > performance of a single TCP stream of about 12MB/sec through the > bridging machine as measured by NetPIPE and netperf. > The bottleneck must be in ng_tee(4) -- the latter uses m_dup(9) when a duplicate is needed, which is very expensive as it has to create a writable copy of the entire mbuf chain (the original chain is DMA'ed into the host memory by the network card). > I'm wondering if bumping the recvspace should help, if changing the > ngsocket hook to queue incoming data should help, if it would be best to > replace ngsocket with a memory-mapped interface, or if anyone has any > other ideas that would help performance. > If you absolutely need to see *all* GigE traffic in userland, then it's going to be troublesome. If not, filter it with ng_bpf(4). Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFCA9+EqRfpzJluFF4RAsT/AJ9iyLMh/ktOtfb8Ko1qt/ApT0oFBQCgiwAZ hoJcv9Tew4LbQtX3zaMdVBo= =q5r3 -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20050204204804.GC71363>