From owner-freebsd-drivers@FreeBSD.ORG Tue Feb 21 21:59:14 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F922106566B for ; Tue, 21 Feb 2012 21:59:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6775A8FC14 for ; Tue, 21 Feb 2012 21:59:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id D749646B2A; Tue, 21 Feb 2012 16:59:13 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 55AF2B94F; Tue, 21 Feb 2012 16:59:13 -0500 (EST) From: John Baldwin To: freebsd-drivers@freebsd.org Date: Tue, 21 Feb 2012 15:04:33 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201202211504.34169.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 21 Feb 2012 16:59:13 -0500 (EST) Cc: Subject: Re: bus_dma coalesce advice X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Feb 2012 21:59:14 -0000 On Monday, February 20, 2012 1:05:40 pm JD Louw wrote: > Hi, > > I have a Xilinx FPGA PCIe DMA design that I'd like to get going on > FreeBSD. I'd like some advice on the best practice of the bus_dma > functions. Specifically, I'd like to understand how best to coalesce > multiple DMA transactions. > > Using the bus_dma_tag_create and bus_dmamem_alloc functions I create > 256 contiguous descriptors. > > bus_dma_tag_create(NULL, /* parent */ > 4, /* alignment */ > 0, /* bounds */ > BUS_SPACE_MAXADDR, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > 256*sizeof(descriptor), /* maxsize */ > 1, /* nsegments */ > 256*sizeof(descriptor), /* maxsegsize */ > BUS_DMA_ALLOCNOW, /* flags */ > NULL, NULL, /* lockfunc, lockarg */ > &desc_tag); /* dmat */ > > I then create another bus_dma_tag for the data: > > bus_dma_tag_create(NULL, /* parent */ > 4, /* alignment */ > 0, /* bounds */ > BUS_SPACE_MAXADDR, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > 0xFFFFF, /* maxsize - 1MB */ > 256, /* nsegments */ > 0x1000, /* maxsegsize - 4KB*/ > BUS_DMA_ALLOCNOW, /* flags */ > NULL, NULL, /* lockfunc, lockarg */ > &data_tag); /* dmat */ > > Now my question: In order to batch several mbufs/uios in into the 256 > descriptors I'd like to do multiple bus_dmamap_loads on the data tag. > But reading the bus_dmamap_load_mbuf/uio code it looks like this is > not a good idea. Each mapping operation does not subtract its nsegment > count from the tag maximum nsegment count, so at some point > bus_dmamap_load will overrun my 256 descriptors. Does your DMA engine really allow a single transaction to span more than 256 descriptors? (The 'nsegmenets' is the maximum number of S/G entries for a single transaction, not the number of entries in your ring.) > Do I need to allocate a separate set of descriptors for each bus_dmamapping? > > Any advice much appreciated, Typically in a NIC driver you will use bus_dmamap_load_mbuf_sg() to populate an array of S/G elements on the stack. You can check the returned value for the number of segments and handle the case where it exceeds the number of segments you actually have available (e.g. by calling m_collapse() or m_defrag() or just queueing the packet until you get a TX completion interrupt that frees up some descriptors). Note that for all of those cases you will need to do a bus_dmamap_unload() first. -- John Baldwin