Date: Fri, 20 Apr 2012 00:03:08 +0200 From: Luigi Rizzo <rizzo@iet.unipi.it> To: Andre Oppermann <andre@freebsd.org> Cc: current@freebsd.org, net@freebsd.org Subject: Re: Some performance measurements on the FreeBSD network stack Message-ID: <20120419220308.GB95692@onelab2.iet.unipi.it> In-Reply-To: <4F908180.6010408@freebsd.org> References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <4F908180.6010408@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Apr 19, 2012 at 11:20:00PM +0200, Andre Oppermann wrote: > On 19.04.2012 22:46, Luigi Rizzo wrote: ... > >What might be moderately expensive are the critical_enter()/critical_exit() > >calls around individual allocations. > > Can't get away from those as a thread must not migrate away > when manipulating the per-CPU mbuf pool. i understand. > >The allocation happens while the code has already an exclusive > >lock on so->snd_buf so a pool of fresh buffers could be attached > >there. > > Ah, there it is not necessary to hold the snd_buf lock while > doing the allocate+copyin. With soreceive_stream() (which is it is not held in the tx path either -- but there is a short section before m_uiotombuf() which does ... SOCKBUF_LOCK(&so->so_snd); // check for pending errors, sbspace, so_state SOCKBUF_UNLOCK(&so->so_snd); ... (some of this is slightly dubious, but that's another story) > >But the other consideration is that one could defer the mbuf allocation > >to a later time when the packet is actually built (or anyways > >right before the thread returns). > >What i envision (and this would fit nicely with netmap) is the following: > >- have a (possibly readonly) template for the headers (MAC+IP+UDP) > > attached to the socket, built on demand, and cached and managed > > with similar invalidation rules as used by fastforward; > > That would require to cross-pointer the rtentry and whatnot again. i was planning to keep a copy, not a reference. If the copy becomes temporarily stale, no big deal, as long as you can detect it reasonably quiclky -- routes are not guaranteed to be correct, anyways. > >- possibly extend the pru_send interface so one can pass down the uio > > instead of the mbuf; > >- make an opportunistic buffer allocation in some place downstream, > > where the code already has an x-lock on some resource (could be > > the snd_buf, the interface, ...) so the allocation comes for free. > > ETOOCOMPLEXOVERTIME. maybe. But i want to investigate this. cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120419220308.GB95692>