From owner-freebsd-drivers@FreeBSD.ORG Mon Feb 20 18:34:21 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4941106564A for ; Mon, 20 Feb 2012 18:34:21 +0000 (UTC) (envelope-from jdl.ntq@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id AE0CB8FC0A for ; Mon, 20 Feb 2012 18:34:21 +0000 (UTC) Received: by iaeo4 with SMTP id o4so10522174iae.13 for ; Mon, 20 Feb 2012 10:34:21 -0800 (PST) Received-SPF: pass (google.com: domain of jdl.ntq@gmail.com designates 10.50.185.228 as permitted sender) client-ip=10.50.185.228; Authentication-Results: mr.google.com; spf=pass (google.com: domain of jdl.ntq@gmail.com designates 10.50.185.228 as permitted sender) smtp.mail=jdl.ntq@gmail.com; dkim=pass header.i=jdl.ntq@gmail.com Received: from mr.google.com ([10.50.185.228]) by 10.50.185.228 with SMTP id ff4mr14795342igc.17.1329762861172 (num_hops = 1); Mon, 20 Feb 2012 10:34:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=Ag0v4U2rwRQgE4dXEGfISwRno1IzSt/k1LTnMioPT/A=; b=gl15iCWbQ5JfAwwNIzjI8+ENjZV03E5cUAud+nSwIXYhJjpPWxjk2s551iXUrBuL/3 JMmb57p0F04e+/pZWFt1YmENoJFCn3o4DIY7i+tYAE1VmycU2qFwiRf2DAV348ZsNVws R9jI6QlCBqcLu4IBUlCXLvJNCcKNvN3FDAgh8= MIME-Version: 1.0 Received: by 10.50.185.228 with SMTP id ff4mr11844183igc.17.1329761140994; Mon, 20 Feb 2012 10:05:40 -0800 (PST) Received: by 10.231.51.18 with HTTP; Mon, 20 Feb 2012 10:05:40 -0800 (PST) Date: Mon, 20 Feb 2012 20:05:40 +0200 Message-ID: From: JD Louw To: freebsd-drivers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: bus_dma coalesce advice X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Feb 2012 18:34:22 -0000 Hi, I have a Xilinx FPGA PCIe DMA design that I'd like to get going on FreeBSD. I'd like some advice on the best practice of the bus_dma functions. Specifically, I'd like to understand how best to coalesce multiple DMA transactions. Using the bus_dma_tag_create and bus_dmamem_alloc functions I create 256 contiguous descriptors. bus_dma_tag_create(NULL, /* parent */ 4, /* alignment */ 0, /* bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ 256*sizeof(descriptor), /* maxsize */ 1, /* nsegments */ 256*sizeof(descriptor), /* maxsegsize */ BUS_DMA_ALLOCNOW, /* flags */ NULL, NULL, /* lockfunc, lockarg */ &desc_tag); /* dmat */ I then create another bus_dma_tag for the data: bus_dma_tag_create(NULL, /* parent */ 4, /* alignment */ 0, /* bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ 0xFFFFF, /* maxsize - 1MB */ 256, /* nsegments */ 0x1000, /* maxsegsize - 4KB*/ BUS_DMA_ALLOCNOW, /* flags */ NULL, NULL, /* lockfunc, lockarg */ &data_tag); /* dmat */ Now my question: In order to batch several mbufs/uios in into the 256 descriptors I'd like to do multiple bus_dmamap_loads on the data tag. But reading the bus_dmamap_load_mbuf/uio code it looks like this is not a good idea. Each mapping operation does not subtract its nsegment count from the tag maximum nsegment count, so at some point bus_dmamap_load will overrun my 256 descriptors. Do I need to allocate a separate set of descriptors for each bus_dmamapping? Any advice much appreciated, JD From owner-freebsd-drivers@FreeBSD.ORG Tue Feb 21 21:59:14 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F922106566B for ; Tue, 21 Feb 2012 21:59:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6775A8FC14 for ; Tue, 21 Feb 2012 21:59:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id D749646B2A; Tue, 21 Feb 2012 16:59:13 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 55AF2B94F; Tue, 21 Feb 2012 16:59:13 -0500 (EST) From: John Baldwin To: freebsd-drivers@freebsd.org Date: Tue, 21 Feb 2012 15:04:33 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201202211504.34169.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 21 Feb 2012 16:59:13 -0500 (EST) Cc: Subject: Re: bus_dma coalesce advice X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Feb 2012 21:59:14 -0000 On Monday, February 20, 2012 1:05:40 pm JD Louw wrote: > Hi, > > I have a Xilinx FPGA PCIe DMA design that I'd like to get going on > FreeBSD. I'd like some advice on the best practice of the bus_dma > functions. Specifically, I'd like to understand how best to coalesce > multiple DMA transactions. > > Using the bus_dma_tag_create and bus_dmamem_alloc functions I create > 256 contiguous descriptors. > > bus_dma_tag_create(NULL, /* parent */ > 4, /* alignment */ > 0, /* bounds */ > BUS_SPACE_MAXADDR, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > 256*sizeof(descriptor), /* maxsize */ > 1, /* nsegments */ > 256*sizeof(descriptor), /* maxsegsize */ > BUS_DMA_ALLOCNOW, /* flags */ > NULL, NULL, /* lockfunc, lockarg */ > &desc_tag); /* dmat */ > > I then create another bus_dma_tag for the data: > > bus_dma_tag_create(NULL, /* parent */ > 4, /* alignment */ > 0, /* bounds */ > BUS_SPACE_MAXADDR, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > 0xFFFFF, /* maxsize - 1MB */ > 256, /* nsegments */ > 0x1000, /* maxsegsize - 4KB*/ > BUS_DMA_ALLOCNOW, /* flags */ > NULL, NULL, /* lockfunc, lockarg */ > &data_tag); /* dmat */ > > Now my question: In order to batch several mbufs/uios in into the 256 > descriptors I'd like to do multiple bus_dmamap_loads on the data tag. > But reading the bus_dmamap_load_mbuf/uio code it looks like this is > not a good idea. Each mapping operation does not subtract its nsegment > count from the tag maximum nsegment count, so at some point > bus_dmamap_load will overrun my 256 descriptors. Does your DMA engine really allow a single transaction to span more than 256 descriptors? (The 'nsegmenets' is the maximum number of S/G entries for a single transaction, not the number of entries in your ring.) > Do I need to allocate a separate set of descriptors for each bus_dmamapping? > > Any advice much appreciated, Typically in a NIC driver you will use bus_dmamap_load_mbuf_sg() to populate an array of S/G elements on the stack. You can check the returned value for the number of segments and handle the case where it exceeds the number of segments you actually have available (e.g. by calling m_collapse() or m_defrag() or just queueing the packet until you get a TX completion interrupt that frees up some descriptors). Note that for all of those cases you will need to do a bus_dmamap_unload() first. -- John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Wed Feb 22 15:06:24 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F25DD106566C for ; Wed, 22 Feb 2012 15:06:24 +0000 (UTC) (envelope-from jdl.ntq@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id AEF508FC0A for ; Wed, 22 Feb 2012 15:06:24 +0000 (UTC) Received: by ghbg15 with SMTP id g15so87474ghb.13 for ; Wed, 22 Feb 2012 07:06:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Iqlx1j9wM9PbmMhaXspoAT+m5YQ8HZl+B0PHOhYn2Nc=; b=HD6Ct+tO+56xT7XEbxbixHeha5k1pB89oy9A5sDiMoz/PPeB2c9UjAHY/LzmwtxJr5 /sl48nrltn7iHabm3v4XsW3lKEUgpLDb6eeuFdSBrJHtyH62RSl0/qBw0laX3dkS4/qI G9zPowtuXVMh7K/O/Ley0RtAADq7DWWL0wEa0= MIME-Version: 1.0 Received: by 10.50.85.227 with SMTP id k3mr22014363igz.17.1329923183861; Wed, 22 Feb 2012 07:06:23 -0800 (PST) Received: by 10.231.51.18 with HTTP; Wed, 22 Feb 2012 07:06:23 -0800 (PST) In-Reply-To: <201202211504.34169.jhb@freebsd.org> References: <201202211504.34169.jhb@freebsd.org> Date: Wed, 22 Feb 2012 17:06:23 +0200 Message-ID: From: JD Louw To: John Baldwin Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-drivers@freebsd.org Subject: Re: bus_dma coalesce advice X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 15:06:25 -0000 On Tue, Feb 21, 2012 at 10:04 PM, John Baldwin wrote: > On Monday, February 20, 2012 1:05:40 pm JD Louw wrote: >> Hi, >> >> I have a Xilinx FPGA PCIe DMA design that I'd like to get going on >> FreeBSD. I'd like some advice on the best practice of the bus_dma >> functions. Specifically, I'd like to understand how best to coalesce >> multiple DMA transactions. >> >> Using the bus_dma_tag_create and bus_dmamem_alloc functions I create >> 256 contiguous descriptors. >> >> =C2=A0 =C2=A0 =C2=A0 bus_dma_tag_create(NULL, =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* parent */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 4, =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/* alignment */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0, =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/* bounds */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_SPACE_MAXADDR, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* lowaddr */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_SPACE_MAXADDR, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* highaddr */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NULL, NULL, =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* filter, filt= erarg */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256*sizeof(descriptor),= =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* maxsize */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1, =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/* nsegments */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256*sizeof(descriptor),= =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* maxsegsize */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_DMA_ALLOCNOW, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* flags */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NULL, NULL, =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* lockfunc, lo= ckarg */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &desc_tag); =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* dmat */ >> >> I then create another bus_dma_tag for the data: >> >> =C2=A0 =C2=A0 =C2=A0 bus_dma_tag_create(NULL, =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* parent */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 4, =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/* alignment */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0, =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0/* bounds */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_SPACE_MAXADDR, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* lowaddr */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_SPACE_MAXADDR, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* highaddr */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NULL, NULL, =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* filter, filt= erarg */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0xFFFFF, =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* max= size - 1MB */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 256, =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0/* nsegments */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0x1000, =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* ma= xsegsize - 4KB*/ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 BUS_DMA_ALLOCNOW, =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* flags */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NULL, NULL, =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* lockfunc, lo= ckarg */ >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &data_tag); =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* dmat */ >> >> Now my question: In order to batch several mbufs/uios in into the 256 >> descriptors I'd like to do multiple bus_dmamap_loads on the data tag. >> But reading the bus_dmamap_load_mbuf/uio code it looks like this is >> not a good idea. Each mapping operation does not subtract its nsegment >> count from the tag maximum nsegment count, so at some point >> bus_dmamap_load will overrun my 256 descriptors. > > Does your DMA engine really allow a single transaction to span more than = 256 > descriptors? =C2=A0(The 'nsegmenets' is the maximum number of S/G entries= for a > single transaction, not the number of entries in your ring.) > >> Do I need to allocate a separate set of descriptors for each bus_dmamapp= ing? >> >> Any advice much appreciated, > > Typically in a NIC driver you will use bus_dmamap_load_mbuf_sg() to popul= ate > an array of S/G elements on the stack. =C2=A0You can check the returned v= alue for > the number of segments and handle the case where it exceeds the number of > segments you actually have available (e.g. by calling m_collapse() or > m_defrag() or just queueing the packet until you get a TX completion inte= rrupt > that frees up some descriptors). =C2=A0Note that for all of those cases y= ou will > need to do a bus_dmamap_unload() first. > > -- > John Baldwin I'm not sure how NIC ring buffer descriptors are structured, but the FPGA DMA descriptor structure looks as follows: struct descriptor { uint64_t seg_phys_addr; uint32_t seg_len; uint64_t next_desc_phys_addr; }; The FPGA's descriptors are chained together in a linked list using physical addressing, so I can chain together as many descriptors as I want to. The chain is terminated by a NULL pointer. Something like this: d------>d------->d------>NULL | | | | | | =E2=8C=84 =E2=8C=84 =E2=8C=84 seg seg seg The physical address of the first descriptor is written to a DMA address register and the engine is started by writing to the DMA control register. A hardware interrupt is generated once the DMA engine is done walking the descriptor chain and sucking in all segments. I'd like lessen the interrupt load by loading more than one uio/mbuf map into the chain before starting off the DMA. But since I don't know beforehand how many segments each uio/mbuf load will occupy I may overrun the 256 chain elements. One solution I can think of is to create multiple smaller descriptor chains (let's say 64 descriptors long), one for each uio/mbuf. Then after loading multiple uio/mbufs I can link the occupied parts of each chain together in one big chain. From owner-freebsd-drivers@FreeBSD.ORG Wed Feb 22 21:35:12 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3BCD1065675 for ; Wed, 22 Feb 2012 21:35:12 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 696BB8FC12 for ; Wed, 22 Feb 2012 21:35:12 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 1C3C846B39; Wed, 22 Feb 2012 16:35:12 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 81F48B9A5; Wed, 22 Feb 2012 16:35:11 -0500 (EST) From: John Baldwin To: JD Louw Date: Wed, 22 Feb 2012 16:31:30 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: <201202211504.34169.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <201202221631.30508.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 22 Feb 2012 16:35:11 -0500 (EST) Cc: freebsd-drivers@freebsd.org Subject: Re: bus_dma coalesce advice X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 21:35:12 -0000 On Wednesday, February 22, 2012 10:06:23 am JD Louw wrote: > On Tue, Feb 21, 2012 at 10:04 PM, John Baldwin wrote: > > On Monday, February 20, 2012 1:05:40 pm JD Louw wrote: > >> Hi, > >> > >> I have a Xilinx FPGA PCIe DMA design that I'd like to get going on > >> FreeBSD. I'd like some advice on the best practice of the bus_dma > >> functions. Specifically, I'd like to understand how best to coalesce > >> multiple DMA transactions. > >> > >> Using the bus_dma_tag_create and bus_dmamem_alloc functions I create > >> 256 contiguous descriptors. > >> > >> bus_dma_tag_create(NULL, /* parent */ > >> 4, /* alignment */ > >> 0, /* bounds */ > >> BUS_SPACE_MAXADDR, /* lowaddr */ > >> BUS_SPACE_MAXADDR, /* highaddr */ > >> NULL, NULL, /* filter, filterarg */ > >> 256*sizeof(descriptor), /* maxsize */ > >> 1, /* nsegments */ > >> 256*sizeof(descriptor), /* maxsegsize */ > >> BUS_DMA_ALLOCNOW, /* flags */ > >> NULL, NULL, /* lockfunc, lockarg */ > >> &desc_tag); /* dmat */ > >> > >> I then create another bus_dma_tag for the data: > >> > >> bus_dma_tag_create(NULL, /* parent */ > >> 4, /* alignment */ > >> 0, /* bounds */ > >> BUS_SPACE_MAXADDR, /* lowaddr */ > >> BUS_SPACE_MAXADDR, /* highaddr */ > >> NULL, NULL, /* filter, filterarg */ > >> 0xFFFFF, /* maxsize - 1MB */ > >> 256, /* nsegments */ > >> 0x1000, /* maxsegsize - 4KB*/ > >> BUS_DMA_ALLOCNOW, /* flags */ > >> NULL, NULL, /* lockfunc, lockarg */ > >> &data_tag); /* dmat */ > >> > >> Now my question: In order to batch several mbufs/uios in into the 256 > >> descriptors I'd like to do multiple bus_dmamap_loads on the data tag. > >> But reading the bus_dmamap_load_mbuf/uio code it looks like this is > >> not a good idea. Each mapping operation does not subtract its nsegment > >> count from the tag maximum nsegment count, so at some point > >> bus_dmamap_load will overrun my 256 descriptors. > > > > Does your DMA engine really allow a single transaction to span more tha= n 256 > > descriptors? (The 'nsegmenets' is the maximum number of S/G entries fo= r a > > single transaction, not the number of entries in your ring.) > > > >> Do I need to allocate a separate set of descriptors for each bus_dmama= pping? > >> > >> Any advice much appreciated, > > > > Typically in a NIC driver you will use bus_dmamap_load_mbuf_sg() to pop= ulate > > an array of S/G elements on the stack. You can check the returned valu= e for > > the number of segments and handle the case where it exceeds the number = of > > segments you actually have available (e.g. by calling m_collapse() or > > m_defrag() or just queueing the packet until you get a TX completion in= terrupt > > that frees up some descriptors). Note that for all of those cases you = will > > need to do a bus_dmamap_unload() first. > > > > -- > > John Baldwin >=20 > I'm not sure how NIC ring buffer descriptors are structured, but the > FPGA DMA descriptor structure looks as follows: >=20 > struct descriptor { > uint64_t seg_phys_addr; > uint32_t seg_len; > uint64_t next_desc_phys_addr; > }; >=20 > The FPGA's descriptors are chained together in a linked list using > physical addressing, so I can chain together as many descriptors as I > want to. The chain is terminated by a NULL pointer. Something like > this: >=20 > d------>d------->d------>NULL > | | | > | | | > =E2=8C=84 =E2=8C=84 =E2=8C=84 > seg seg seg >=20 >=20 > The physical address of the first descriptor is written to a DMA > address register and the engine is started by writing to the DMA > control register. A hardware interrupt is generated once the DMA > engine is done walking the descriptor chain and sucking in all > segments. >=20 >=20 > I'd like lessen the interrupt load by loading more than one uio/mbuf > map into the chain before starting off the DMA. But since I don't know > beforehand how many segments each uio/mbuf load will occupy I may > overrun the 256 chain elements. Yes, this is the same problem. Just keep loading packets on until you either run out of mbufs, or you have an mbuf that has too many segments. If the mbuf has too many segments, you put it back in the IFQ so that it will be retried once you get an interrupt from your card. Just keep track of your current pending global chain you are building in your softc. =2D-=20 John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Wed Feb 22 22:47:40 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF926106566C for ; Wed, 22 Feb 2012 22:47:40 +0000 (UTC) (envelope-from adarsh.joshi@qlogic.com) Received: from AM1EHSOBE006.bigfish.com (am1ehsobe004.messaging.microsoft.com [213.199.154.207]) by mx1.freebsd.org (Postfix) with ESMTP id 40DF78FC08 for ; Wed, 22 Feb 2012 22:47:39 +0000 (UTC) Received: from mail111-am1-R.bigfish.com (10.3.201.244) by AM1EHSOBE006.bigfish.com (10.3.204.26) with Microsoft SMTP Server id 14.1.225.23; Wed, 22 Feb 2012 22:32:27 +0000 Received: from mail111-am1 (localhost [127.0.0.1]) by mail111-am1-R.bigfish.com (Postfix) with ESMTP id 45DF51805BD for ; Wed, 22 Feb 2012 22:32:27 +0000 (UTC) X-SpamScore: 1 X-BigFish: VPS1(zzc85fhzz1202hzz8275bh8275dhz2ei2a8h668h839h) X-Forefront-Antispam-Report: CIP:198.70.193.64; KIP:(null); UIP:(null); IPV:NLI; H:avexcashub1.qlogic.com; RD:avexcashub2.qlogic.com; EFVD:NLI Received-SPF: neutral (mail111-am1: 198.70.193.64 is neither permitted nor denied by domain of qlogic.com) client-ip=198.70.193.64; envelope-from=adarsh.joshi@qlogic.com; helo=avexcashub1.qlogic.com ; 1.qlogic.com ; Received: from mail111-am1 (localhost.localdomain [127.0.0.1]) by mail111-am1 (MessageSwitch) id 1329949945150197_5641; Wed, 22 Feb 2012 22:32:25 +0000 (UTC) Received: from AM1EHSMHS007.bigfish.com (unknown [10.3.201.248]) by mail111-am1.bigfish.com (Postfix) with ESMTP id 16C31C0043 for ; Wed, 22 Feb 2012 22:32:25 +0000 (UTC) Received: from avexcashub1.qlogic.com (198.70.193.64) by AM1EHSMHS007.bigfish.com (10.3.207.107) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 22 Feb 2012 22:32:24 +0000 Received: from avexmb1.qlogic.org ([fe80::9545:3a4f:c131:467d]) by avexcashub2.qlogic.org ([::1]) with mapi; Wed, 22 Feb 2012 14:32:27 -0800 From: Adarsh Joshi To: "freebsd-drivers@freebsd.org" Date: Wed, 22 Feb 2012 14:32:27 -0800 Thread-Topic: Question on SOCKBUF_DEBUG option Thread-Index: AczxsYRqs56SBP3YR6WGFMna6BirsQ== Message-ID: <5E4F49720D0BAD499EE1F01232234BA87437EBA4BC@AVEXMB1.qlogic.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-OriginatorOrg: qlogic.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Question on SOCKBUF_DEBUG option X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 22:47:41 -0000 Hi, I read that the SOCKBUF_DEBUG option performs runtime consistency checks on= socket buffers. I am interested to know more about it. Can someone explain how this works and what information I can expect from i= t? Thanks regards Adarsh ________________________________ This message and any attached documents contain information from QLogic Cor= poration or its wholly-owned subsidiaries that may be confidential. If you = are not the intended recipient, you may not read, copy, distribute, or use = this information. If you have received this transmission in error, please n= otify the sender immediately by reply e-mail and then delete this message. From owner-freebsd-drivers@FreeBSD.ORG Thu Feb 23 20:43:50 2012 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CD29106564A for ; Thu, 23 Feb 2012 20:43:50 +0000 (UTC) (envelope-from adarsh.joshi@qlogic.com) Received: from VA3EHSOBE008.bigfish.com (va3ehsobe001.messaging.microsoft.com [216.32.180.11]) by mx1.freebsd.org (Postfix) with ESMTP id 03B2C8FC12 for ; Thu, 23 Feb 2012 20:43:49 +0000 (UTC) Received: from mail44-va3-R.bigfish.com (10.7.14.237) by VA3EHSOBE008.bigfish.com (10.7.40.28) with Microsoft SMTP Server id 14.1.225.23; Thu, 23 Feb 2012 20:43:48 +0000 Received: from mail44-va3 (localhost [127.0.0.1]) by mail44-va3-R.bigfish.com (Postfix) with ESMTP id 8B88CC03EB for ; Thu, 23 Feb 2012 20:43:48 +0000 (UTC) X-SpamScore: 0 X-BigFish: VPS0(zz9371Ic85fhzz1202hzz8275bh8275dhz2ei2a8h668h839h) X-Forefront-Antispam-Report: CIP:198.70.193.64; KIP:(null); UIP:(null); IPV:NLI; H:avexcashub1.qlogic.com; RD:avexcashub2.qlogic.com; EFVD:NLI Received-SPF: neutral (mail44-va3: 198.70.193.64 is neither permitted nor denied by domain of qlogic.com) client-ip=198.70.193.64; envelope-from=adarsh.joshi@qlogic.com; helo=avexcashub1.qlogic.com ; 1.qlogic.com ; Received: from mail44-va3 (localhost.localdomain [127.0.0.1]) by mail44-va3 (MessageSwitch) id 1330029826216051_28321; Thu, 23 Feb 2012 20:43:46 +0000 (UTC) Received: from VA3EHSMHS012.bigfish.com (unknown [10.7.14.242]) by mail44-va3.bigfish.com (Postfix) with ESMTP id 2FC463800C2 for ; Thu, 23 Feb 2012 20:43:46 +0000 (UTC) Received: from avexcashub1.qlogic.com (198.70.193.64) by VA3EHSMHS012.bigfish.com (10.7.99.22) with Microsoft SMTP Server (TLS) id 14.1.225.23; Thu, 23 Feb 2012 20:43:45 +0000 Received: from avexmb1.qlogic.org ([fe80::9545:3a4f:c131:467d]) by avexcashub2.qlogic.org ([::1]) with mapi; Thu, 23 Feb 2012 12:43:44 -0800 From: Adarsh Joshi To: "freebsd-drivers@freebsd.org" Date: Thu, 23 Feb 2012 12:43:42 -0800 Thread-Topic: Question on SOCKBUF_DEBUG option Thread-Index: AczxsYRqs56SBP3YR6WGFMna6BirsQAufuDw Message-ID: <5E4F49720D0BAD499EE1F01232234BA87437EBA678@AVEXMB1.qlogic.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US MIME-Version: 1.0 X-OriginatorOrg: qlogic.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: RE: Question on SOCKBUF_DEBUG option X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 20:43:50 -0000 Hello again, I am trying to debug socket timeout issues during my stress tests. Can anyo= ne suggest any debug options or flags I can turn on to get more information= ? Thanks a lot Adarsh From: Adarsh Joshi Sent: Wednesday, February 22, 2012 2:32 PM To: freebsd-drivers@freebsd.org Subject: Question on SOCKBUF_DEBUG option Hi, I read that the SOCKBUF_DEBUG option performs runtime consistency checks on= socket buffers. I am interested to know more about it. Can someone explain how this works and what information I can expect from i= t? Thanks regards Adarsh ________________________________ This message and any attached documents contain information from QLogic Cor= poration or its wholly-owned subsidiaries that may be confidential. If you = are not the intended recipient, you may not read, copy, distribute, or use = this information. If you have received this transmission in error, please n= otify the sender immediately by reply e-mail and then delete this message.