From owner-freebsd-current@FreeBSD.ORG Thu Apr 19 21:43:37 2012 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 039961065679; Thu, 19 Apr 2012 21:43:37 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id B64F68FC12; Thu, 19 Apr 2012 21:43:36 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 0F42873027; Fri, 20 Apr 2012 00:03:08 +0200 (CEST) Date: Fri, 20 Apr 2012 00:03:08 +0200 From: Luigi Rizzo To: Andre Oppermann Message-ID: <20120419220308.GB95692@onelab2.iet.unipi.it> References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <4F908180.6010408@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4F908180.6010408@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: current@freebsd.org, net@freebsd.org Subject: Re: Some performance measurements on the FreeBSD network stack X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2012 21:43:37 -0000 On Thu, Apr 19, 2012 at 11:20:00PM +0200, Andre Oppermann wrote: > On 19.04.2012 22:46, Luigi Rizzo wrote: ... > >What might be moderately expensive are the critical_enter()/critical_exit() > >calls around individual allocations. > > Can't get away from those as a thread must not migrate away > when manipulating the per-CPU mbuf pool. i understand. > >The allocation happens while the code has already an exclusive > >lock on so->snd_buf so a pool of fresh buffers could be attached > >there. > > Ah, there it is not necessary to hold the snd_buf lock while > doing the allocate+copyin. With soreceive_stream() (which is it is not held in the tx path either -- but there is a short section before m_uiotombuf() which does ... SOCKBUF_LOCK(&so->so_snd); // check for pending errors, sbspace, so_state SOCKBUF_UNLOCK(&so->so_snd); ... (some of this is slightly dubious, but that's another story) > >But the other consideration is that one could defer the mbuf allocation > >to a later time when the packet is actually built (or anyways > >right before the thread returns). > >What i envision (and this would fit nicely with netmap) is the following: > >- have a (possibly readonly) template for the headers (MAC+IP+UDP) > > attached to the socket, built on demand, and cached and managed > > with similar invalidation rules as used by fastforward; > > That would require to cross-pointer the rtentry and whatnot again. i was planning to keep a copy, not a reference. If the copy becomes temporarily stale, no big deal, as long as you can detect it reasonably quiclky -- routes are not guaranteed to be correct, anyways. > >- possibly extend the pru_send interface so one can pass down the uio > > instead of the mbuf; > >- make an opportunistic buffer allocation in some place downstream, > > where the code already has an x-lock on some resource (could be > > the snd_buf, the interface, ...) so the allocation comes for free. > > ETOOCOMPLEXOVERTIME. maybe. But i want to investigate this. cheers luigi