From owner-freebsd-current@FreeBSD.ORG Fri Sep 29 23:11:02 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 187AB16A40F; Fri, 29 Sep 2006 23:11:02 +0000 (UTC) (envelope-from rrs@cisco.com) Received: from sj-iport-1.cisco.com (sj-iport-1-in.cisco.com [171.71.176.70]) by mx1.FreeBSD.org (Postfix) with ESMTP id 953B443D6E; Fri, 29 Sep 2006 23:10:48 +0000 (GMT) (envelope-from rrs@cisco.com) Received: from sj-dkim-2.cisco.com ([171.71.179.186]) by sj-iport-1.cisco.com with ESMTP; 29 Sep 2006 16:10:49 -0700 Received: from sj-core-5.cisco.com (sj-core-5.cisco.com [171.71.177.238]) by sj-dkim-2.cisco.com (8.12.11.20060308/8.12.11) with ESMTP id k8TNAmad020013; Fri, 29 Sep 2006 16:10:48 -0700 Received: from xbh-sjc-211.amer.cisco.com (xbh-sjc-211.cisco.com [171.70.151.144]) by sj-core-5.cisco.com (8.12.10/8.12.6) with ESMTP id k8TNAm1E002493; Fri, 29 Sep 2006 16:10:48 -0700 (PDT) Received: from xfe-sjc-211.amer.cisco.com ([171.70.151.174]) by xbh-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 29 Sep 2006 16:10:48 -0700 Received: from [127.0.0.1] ([171.68.225.134]) by xfe-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 29 Sep 2006 16:10:47 -0700 Message-ID: <451DA7D4.8020609@cisco.com> Date: Fri, 29 Sep 2006 19:10:12 -0400 From: Randall Stewart User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050920 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Andrew Gallatin References: <451C4850.5030302@freebsd.org> <451D884F.1030807@cisco.com> <20060929213722.GR80527@funkthat.com> <451D9440.6060105@cisco.com> <17693.39106.950631.742167@grasshopper.cs.duke.edu> <451D9E59.9050000@freebsd.org> <17693.41475.778558.381395@grasshopper.cs.duke.edu> In-Reply-To: <17693.41475.778558.381395@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 29 Sep 2006 23:10:48.0047 (UTC) FILETIME=[7FC4F7F0:01C6E41C] DKIM-Signature: a=rsa-sha1; q=dns; l=3066; t=1159571448; x=1160435448; c=relaxed/simple; s=sjdkim2002; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=rrs@cisco.com; z=From:Randall=20Stewart=20 |Subject:Re=3A=20Much=20improved=20sosend_*()=20functions; X=v=3Dcisco.com=3B=20h=3DVUE9hWftOvPULvUJ9EhsWe/RDnk=3D; b=y+8iQMsAwHjX3/hVFa2gmkETnLnTAY9ShpyIHn8YCOwcn9AejGSfhIcaA83FYacAstz3KEtC IFScwUU92DBHV9hTg5VHjK37zPDFKXpJt1uGUxWQkPQ+ZZsXrrvJpk8V; Authentication-Results: sj-dkim-2.cisco.com; header.From=rrs@cisco.com; dkim=pass ( sig from cisco.com verified; ); Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, Andre Oppermann Subject: Re: Much improved sosend_*() functions X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Sep 2006 23:11:02 -0000 Andrew Gallatin wrote: > Andre Oppermann writes: > > Andrew Gallatin wrote: > > > Andre, > > > > > > I meant to ask: Did you try 16KB jumbos? Did they perform > > > any better than page-sized jumbos? > > > > No, I didn't try 16K jumbos. The problem with anything larger than > > page size is that it may look contigous in kernel memory but isn't > > in physical memory. Thus you need the same number of descriptors > > for the network card as with page sized (4K) clusters. > > But it would allow you to do one copyin, rather than 4. I > don't know how much this would help, but it might be worth > looking at. It helped the SCTP code quite a bit when I optimized it to use this... can't remember how much of a boost it got.. (I started using all the frames at one time).. and of course it only helps when the msg size being sent is > 9k... But it was a help... at least on the copy-in side for sending down.. > > > > Also, if we're going to change how mbufs work, let's add something > > > like Linux's skb_frag_t frags[MAX_SKB_FRAGS]; In FreeBSD parlence, > > > this embeds something like an array of sf_bufs pointers in mbuf. The > > > big difference to a chain of M_EXT mbufs is that you need to allocate > > > only one mbuf wrapper, rather than one for each item in the list. > > > Also, the reference is kept in the page (or sf_buf) itself, and the > > > data offset is kept in the skbbuf (or mbuf). > > > > We are not going to change how mbufs work. > > > > > This allows us to do cool things like allocate a single page, and use > > > both halves of it for 2 separate 1500 byte frames. This allows us to > > > achieve *amazing* results in combination with LRO, because it allows > > > us to do, on average, many fewer allocations per byte. Especially in > > > combination with Linux's "high order" page allocations. Using order-2 > > > allocations and LRO, I've actually seen 10GbE line rate receives on a > > > wimpy 2.0GHz Athlon64. > > > > I have just started tackling the receive path. Lets see what comes out > > of it first before we jump to conclusions. > > It could be mbufs are cheaper to get than skbs and pages on linux, > but I doubt it. FWIW, linux has an skb chaining mechanism > (frag_list). My first LRO experiment was based on allocating "normal" > skbs and chaining them. That maxed out at around 5.2Gb/s (on the same > hardware I see line rate on). This would be a drastic set of changes.. a bit more than the simple add a few more sizes and get rid of data inside the mbuf... which would shrink the mbuf size considerable... of course one would need always a data EXT... R > > Drew > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- Randall Stewart NSSTG - Cisco Systems Inc. 803-345-0369 815-342-5222 (cell)