From owner-freebsd-current@FreeBSD.ORG Thu Apr 19 22:37:28 2012 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFE0C106566B for ; Thu, 19 Apr 2012 22:37:28 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1E0F38FC18 for ; Thu, 19 Apr 2012 22:37:27 +0000 (UTC) Received: (qmail 16058 invoked from network); 19 Apr 2012 22:32:39 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 19 Apr 2012 22:32:39 -0000 Message-ID: <4F9093A1.3080305@freebsd.org> Date: Fri, 20 Apr 2012 00:37:21 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20120327 Thunderbird/11.0.1 MIME-Version: 1.0 To: Luigi Rizzo References: <20120419133018.GA91364@onelab2.iet.unipi.it> <4F907011.9080602@freebsd.org> <20120419204622.GA94904@onelab2.iet.unipi.it> <4F908180.6010408@freebsd.org> <20120419220308.GB95692@onelab2.iet.unipi.it> In-Reply-To: <20120419220308.GB95692@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org, net@freebsd.org Subject: Re: Some performance measurements on the FreeBSD network stack X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2012 22:37:28 -0000 On 20.04.2012 00:03, Luigi Rizzo wrote: > On Thu, Apr 19, 2012 at 11:20:00PM +0200, Andre Oppermann wrote: >> On 19.04.2012 22:46, Luigi Rizzo wrote: >>> The allocation happens while the code has already an exclusive >>> lock on so->snd_buf so a pool of fresh buffers could be attached >>> there. >> >> Ah, there it is not necessary to hold the snd_buf lock while >> doing the allocate+copyin. With soreceive_stream() (which is > > it is not held in the tx path either -- but there is a short section > before m_uiotombuf() which does > > ... > SOCKBUF_LOCK(&so->so_snd); > // check for pending errors, sbspace, so_state > SOCKBUF_UNLOCK(&so->so_snd); > ... > > (some of this is slightly dubious, but that's another story) Indeed the lock isn't held across the m_uiotombuf(). You're talking about filling an sockbuf mbuf cache while holding the lock? >>> But the other consideration is that one could defer the mbuf allocation >>> to a later time when the packet is actually built (or anyways >>> right before the thread returns). >>> What i envision (and this would fit nicely with netmap) is the following: >>> - have a (possibly readonly) template for the headers (MAC+IP+UDP) >>> attached to the socket, built on demand, and cached and managed >>> with similar invalidation rules as used by fastforward; >> >> That would require to cross-pointer the rtentry and whatnot again. > > i was planning to keep a copy, not a reference. If the copy becomes > temporarily stale, no big deal, as long as you can detect it reasonably > quiclky -- routes are not guaranteed to be correct, anyways. Be wary of disappearing interface pointers... >>> - possibly extend the pru_send interface so one can pass down the uio >>> instead of the mbuf; >>> - make an opportunistic buffer allocation in some place downstream, >>> where the code already has an x-lock on some resource (could be >>> the snd_buf, the interface, ...) so the allocation comes for free. >> >> ETOOCOMPLEXOVERTIME. > > maybe. But i want to investigate this. I fail see what passing down the uio would gain you. The snd_buf lock isn't obtained again after the copyin. Not that I want to prevent you from investigating other ways. ;) -- Andre