From owner-freebsd-arch@FreeBSD.ORG Fri Jun 27 14:52:37 2003 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4282A37B405 for ; Fri, 27 Jun 2003 14:52:37 -0700 (PDT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5D5EC43FFD for ; Fri, 27 Jun 2003 14:52:36 -0700 (PDT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.9/8.12.9) with ESMTP id h5RLqYwV005791 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Fri, 27 Jun 2003 17:52:34 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id h5RLqTh00773; Fri, 27 Jun 2003 17:52:29 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16124.48285.343025.428957@grasshopper.cs.duke.edu> Date: Fri, 27 Jun 2003 17:52:29 -0400 (EDT) To: Scott Long In-Reply-To: <3EFCB725.4060902@btc.adaptec.com> References: <3EF3C12F.9060303@btc.adaptec.com> <16124.39930.142492.356163@grasshopper.cs.duke.edu> <3EFC9F2D.6020908@btc.adaptec.com> <16124.43999.333761.397624@grasshopper.cs.duke.edu> <3EFCAC7A.6060305@btc.adaptec.com> <16124.45051.919899.414795@grasshopper.cs.duke.edu> <3EFCB178.9030207@btc.adaptec.com> <16124.46454.595892.860118@grasshopper.cs.duke.edu> <3EFCB725.4060902@btc.adaptec.com> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: Andrew Gallatin cc: freebsd-arch@freebsd.org Subject: Re: API change for bus_dma X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jun 2003 21:52:37 -0000 Scott Long writes: > > > > Unfortunately, in our application we must assume coherency in some > > situations. We have kernel memory mmap'ed into user space for > > zero-copy io of small messages. Doing a system call to force the dma > > sync would add unacceptable latency. (we're talking sub 10us latencies > > here, without syscalls). > > > > The bus_dmamap_sync() isn't done from a separate system call. The flow > is this: > > > bus_dmamap_load(); > driver_callback() > { > set up S/G list; > bus_dmamap_sync(PREWRITE); > tell hardware that DMA is ready; > } > > The callback gets called immediately as long as conditions are met, as > we have discuss prior. > > Then: > > driver_intr() > { > see that hardware has DMA'd data to us; > bus_dmamap_sync(POSTREAD); > bus_dmamap_unload(); > } In our application, millions of separate DMAs can happen with no kernel intervention at all. We do the bus_dmamap_load() in the context of an ioctl which allocates some kernel memory, and pokes the DMA addresses for the kernel memory down onto the board. The user then mmaps the kernel memory, and also mmaps a page of SRAM on our board. When the user wants to initate a dma, he writes to the device's SRAM indicating an offset in the mmaped kernel memory. The board then initates the transfer (after doing bounds checking..). When the user is done, or the process exits, the dma addresses are cleared from the nic, and the kernel memory is freed. This is OS-bypass networking. > As I understand it, it is possible to set the pycho bridge to use > a coherent address range, but FreeBSD doesn't take advantage of that > yet. > Yes, that's what solaris does.. Drew