From owner-freebsd-net@FreeBSD.ORG Wed Apr 8 13:05:10 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 133F0106574E for ; Wed, 8 Apr 2009 13:05:10 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63906.mail.re1.yahoo.com (web63906.mail.re1.yahoo.com [69.147.97.121]) by mx1.freebsd.org (Postfix) with SMTP id C3DFE8FC26 for ; Wed, 8 Apr 2009 13:05:09 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 36277 invoked by uid 60001); 8 Apr 2009 13:05:09 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239195908; bh=YW2tnSmfnygNnRQuQwrrkN1nB3LAe25kxMHcXnYqLwc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=mwRcjdSNzX2t6tZ0kuGlVSJecCzCc/jYyYSVQHR6IpUes0I90Ughtsf+fI2I60qZ97s/MUqMFBtP3cSW7Q/tOsWzi79YifUnEGe30ltB63BX12lAZwTmh+4WZ6X0vEzGtH64V4nsRios4HfkOuXdu7Scx352mK707mhw+bBWNdw= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=1QdE1+CIrbkqa+DN1kIZiyigAAKGWT7Mc8Wi395VMWLLC/Mv+hx4YuUry67M/lYqNLNUpy5Wnjn0Pvcb25mJdXvAofRDv7AK76jQjVtExJt5eMAiYSRK6dVyWu76zR76GrBJUw9DWJX7HTSc/fe0w7odWeeIr70K7DuaYiz3LTk=; Message-ID: <871699.35154.qm@web63906.mail.re1.yahoo.com> X-YMail-OSG: t3W3E6AVM1lR2mfu51oHbNoGyO4mKfzC02dDp96njH5sc6Pr15G275IBpIoPFDtN_WILwo9orGJg7lYKsrzlY_MjrmjZhIzdilc9Li3sutmd2cGt3ExkRAAsG0xEXRc1yi0e3hkmu0EC_WOh8PsrVx8Li.ahudCpkJbPQzOaY6PaVdkQMHwSMEy4h6q7F5NpIuIFLDgrP9jNBrwOIbbZDVHqv_54pWHbolJ15nKOderXTTt5bktGAtF6TLRcrsuxH8vv6w7WkLCMPtxrthhr.UXaBF9RCiTTomEHZeOG.vsrHp7JhAH_rxRioQk8 Received: from [98.242.222.229] by web63906.mail.re1.yahoo.com via HTTP; Wed, 08 Apr 2009 06:05:08 PDT X-Mailer: YahooMailWebService/0.7.289.1 Date: Wed, 8 Apr 2009 06:05:08 -0700 (PDT) From: Barney Cordoba To: Ivan Voras , Robert Watson In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org Subject: Re: Advice on a multithreaded netisr patch? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: barney_cordoba@yahoo.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2009 13:05:10 -0000 --- On Mon, 4/6/09, Robert Watson wrote: > From: Robert Watson > Subject: Re: Advice on a multithreaded netisr patch? > To: "Ivan Voras" > Cc: freebsd-net@freebsd.org > Date: Monday, April 6, 2009, 2:52 PM > On Mon, 6 Apr 2009, Ivan Voras wrote: > > >> I think we're talking slightly at cross > purposes. There are two > >> transfers of interest: > >> > >> (1) DMA of the packet data to main memory from the > NIC > >> (2) Servicing of CPU cache misses to access data > in main memory > >> > >> By the time you receive an interrupt, the DMA is > complete, so once you > > > > OK, this was what was confusing me - for a moment I > thought you meant it's not so. > > It's a polite lie that we will choose to believe the > purposes of simplification. And probably true for all our > drivers in practice right now. > > >> m = m_pullup(m, sizeof(*w)); > >> if (m == NULL) > >> return; > >> w = mtod(m, struct whatever *); > >> > >> m_pullup() here ensures that the first sizeof(*w) > bytes of mbuf data are contiguously stored so that the cast > of w to m's data will point at a > > > > So, m_pullup() can resize / realloc() the mbuf? (not > that it matters for this purpose) > > Yes -- if it can't meet the contiguity requirements > using the current mbuf chain, it may reallocate and return a > new head to the chain (hence m being reassigned). If that > reallocation fails, it may return NULL. Once you've > called m_pullup(), existing pointers into the chain's > data will be invalid, so if you've already called mtod() > on it, you need to call it again. > > >> - A TCP segment will need to be ACK'd, so if > you're sending data in > >> chunks in > >> one direction, the ACKs will not be piggy-backed > on existing data > >> tranfers, > >> and instead be sent independently, hitting the > network stack two more > >> times. > > > > No combination of these can make an accounting > difference between 1,000 and 250,000 pps. I must be hitting > something very bad here. > > Yes, you definitely want to run tcpdump to see what's > going on here. > > >> - Remember that TCP works to expand its window, > and then maintains the > >> highest > >> performance it can by bumping up against the top > of available bandwidth > >> continuously. This involves detecting buffer > limits by generating > >> packets > >> that can't be sent, adding to the packet > count. With loopback > >> traffic, the > >> drop point occurs when you exceed the size of > the netisr's queue for > >> IP, so > >> you might try bumping that from the default to > something much larger. Robert, Is there any work being done on lighter weight locks for queues? It seems ridiculous to avoid using queues because of lock contention when the locks are only protecting a couple lines of code. Barney