From owner-freebsd-net Fri Aug 9 14:33:30 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D4B3937B400; Fri, 9 Aug 2002 14:33:26 -0700 (PDT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 15E4A43E6A; Fri, 9 Aug 2002 14:33:26 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id RAA11900; Fri, 9 Aug 2002 17:33:25 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g79LWtw05379; Fri, 9 Aug 2002 17:32:55 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15700.13575.470677.723138@grasshopper.cs.duke.edu> Date: Fri, 9 Aug 2002 17:32:55 -0400 (EDT) To: Bosko Milekic Cc: ken@freebsd.org, freebsd-net@freebsd.org Subject: Re: Jumbo Clusters in mb_alloc In-Reply-To: <20020809164136.A88798@unixdaemons.com> References: <20020809151627.A88180@unixdaemons.com> <15700.9669.885383.727182@grasshopper.cs.duke.edu> <20020809164136.A88798@unixdaemons.com> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Bosko Milekic writes: > > On Fri, Aug 09, 2002 at 04:27:49PM -0400, Andrew Gallatin wrote: > > > > Bosko Milekic writes: > > > > > > Hi guys, > > > > > > Can you put this (the attached) to use? > > > > > > It's the implementation of jumbo clusters within mb_alloc, as > > > discussed on -net not too long ago. > > > > > > I have not written the allocation interface yet as I would like to > > > know if you could put them to use in some of the drivers, and also > > > what you would need from the interface. > > > > I can certainly use this in my (3rd party) GM Myrinet driver. > > > > I think the tigon driver can be made to use this, but I have no tigon > > hardware anymore. I also think bge could be made to use this, but > > again, no hardware to play with it on. > > Excellent. > > So the question is what do you think would be a convenient interface? > This really boils down to how you plan to use it. I'm assuming that, > for example, you can do scatter DMA to each page (each buffer spans > /at most/ 3 pages, and only 2 pages on alpha) in the driver receive > code. So, assuming that's the case, we can provide an allocation > routine, something like m_getjmb() that will give you an mbuf and a > jumbo buf in one shot. Then you can extract the physical addresses > from the parts of the data region and scatter DMA to it. The buffers > with the mbuf would be easily freed with m_freem(). I already have code to extract physical address from an mbuf in such a way as to make sure that DMA does not cross page boundaries, but I'd hate to see this code replicated everywhere. Ideally, I'd love to see an interface where I call a function dma_map_mbuf_chain (m, ...) and get back an array of DMA addresses. Hmm.. WWNBD (what would NetBSD do..). Ah, yes: bus_dmamap_load_mbuf(sc->sc_dmat, dmamap, m0, BUS_DMA_WRITE|BUS_DMA_NOWAIT) != 0) { MGETHDR(m, M_DONTWAIT, MT_DATA); Methinks its high time we adopted this interface. > What about the send case, though? How do you want to determine The send case opens a huge can-of-worms. Lets say that somebody is forwarding between myrinet & GigE using FreeBSD, with myri0 on the myrinet network, and em0 on GigE. Let's assume that myri0 is using jumbo buffers, but em0 isn't jumbo aware & cannot accept them & the forwarded packets end up as garbage. Perhaps we need to add an IFCAP_JUMBOAWARE flag and somwhere (IF_DEQUE?) add a KASSERT to catch situaions like this. Or even.. ick, add code to copy to a normal mbuf chain. Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message