Date: Wed, 19 Dec 2018 15:15:44 +0530 From: Rajesh Kumar <rajfbsd@gmail.com> To: "O'Connor, Daniel" <darius@dons.net.au> Cc: cem@freebsd.org, FreeBSD Hackers <freebsd-hackers@freebsd.org>, freebsd-drivers@freebsd.org, asomers@freebsd.org Subject: Re: How to use the DMA Engine in FreeBSD? Message-ID: <CAAO%2BANN1e2Pwy1KsssQqy92_ivU_CKtO9wnd=EjmDtSLnw%2BawA@mail.gmail.com> In-Reply-To: <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> References: <CAAO%2BANOty8FeOR7VQdXmxWy5y288pm=Q4cswHJ1BbgT2h1cUWg@mail.gmail.com> <CANCZdfrEv9o6_ydp9Qe73O1v6NqHFtivHpw4m2PKhTyAVaPR9g@mail.gmail.com> <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> <CAOtMX2jXMz3D8_JvOk4M2DrEZDP8sztgy2bdBuv-yNwz9zR2JQ@mail.gmail.com> <CAG6CVpV5hL%2Bk3OcJZmMfvJnSPo5wU1w8o0A85TbBAU49a661zA@mail.gmail.com> <CAAO%2BANPa0uTvZPbq%2BM%2BX1ozA5ytUoAG6RwXhyM5dGzq1ZasFiw@mail.gmail.com> <CAAO%2BANNHHJpMMgXuDSC7ftpTwPr1XPj9_ioYoA83WJooiELwog@mail.gmail.com> <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks for your explanation Daniel. I will see how to initiate the DMA transfer in my case after the bus_dma API's has done its part. On Tue, Dec 18, 2018 at 6:01 PM O'Connor, Daniel <darius@dons.net.au> wrote: > > > > On 18 Dec 2018, at 05:29, Rajesh Kumar <rajfbsd@gmail.com> wrote: > > Any thoughts (or) suggestions here? > > You don't need to "assign DMA channels" and a PCIe device doesn't "master > the bus" because it's all point to point. > > "DMA channels" are a legacy of ISA cards, although the terminology may be > reused for things like ioat. > > For PCI[e] devices which support bus mastering (ie virtually all of them) > the device itself issues the transfer. How that is actually done depends on > the device - usually you program registers with a scatter/gather (SG) list, > or with the physical address of an SG list in system memory and then the > device will do the job. > > The bus_dma* routines in the kernel are a tool box to let a driver tell > the kernel what kind of DMA a card is going to do and to help with things > like mapping it as necessary, cache coherency etc etc. > > Systems with ioat are pretty rare I believe, the vast majority of DMA in a > modern PC is done by PCI cards (or things which look like PCI cards to the > OS). > > > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar <rajfbsd@gmail.com> wrote: > > > >> Hi, > >> > >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to > a > >> peripheral device being made as the owner of the bus by the DMA engine > and > >> do data transfer. Please correct me if I am wrong. But, will this > bus_dma > >> API uses the DMA engine in hardware (does it support PCI busmastering)? > >> > >> As I understand, DMA engine will expose DMA channels, which can be used > by > >> the client drivers to queue their DMA requests (without CPU > intervention) > >> and completions are given through the callback functions. If this is > >> considered the real DMA operation, I don't see options to request DMA > >> channels using bus_dma API's, and the callbacks in bus_dma API is meant > to > >> receive just the DMA map details (doesn't indicate a DMA completion). > >> ioat(4) driver seems to be doing what I described above, but that seems > to > >> be Intel specific. Can it be used with any DMA controllers? > >> > >> So, I am wondering what needs to be done in FreeBSD to do the actual DMA > >> involving DMA engine in a more generic way? > >> > >> Thanks, > >> Rajesh. > >> > >> > >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer <cem@freebsd.org> wrote: > >> > >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <asomers@freebsd.org> > wrote: > >>>>> For some Intel based server hardware, there is the "ioat" driver, > >>> which > >>>>> allows for user code to schedule DMA operations. See ioat(4) for > >>>>> details, including a pointer to the test program. > >>> > >>> This isn't true (or at least, only for testing). It's only usable by > >>> the kernel at the moment. > >>> > >>>> * In what context are callbacks called? Are they called from a signal > >>>> handler, or in a separate thread, or something else? > >>> > >>> Directly from the ithread, mostly. Callers can't take sleep locks or > >>> do very much besides set a flag and wakeup, or kick off a callout or > >>> taskqueue. > >>> > >>>> * Why isn't ioat.h installed? > >>> > >>> Why would it be? It's a kernel interface. > >>> > >>>> * Are "interrupts" synonymous with callbacks? > >>> > >>> I don't understand the question. Interrupts are synonymous with > >>> interrupts. Like, MSI-X. > >>> > >>>> * Do you have a rough idea for about the minimum buffer size that > >>>> makes sense to use with ioat? > >>> > >>> What makes sense will depend on your use case. It's not necessarily > >>> faster than CPU memcpy, but it may be worth some slowdown to offload > >>> latency-insensitive bulk copies. (It may make a lot of sense for > >>> asynchronous page zero, if someone wants to bring that back.) It's > >>> worth benchmarking your specific model. > >>> > >>> We use it for >= 8kB copies on Broadwell, although that's largely an > >>> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes, > >>> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with > >>> some small number of lanes on Broadwell, although I may be mistaken; > >>> and it may not communicate with RAM over the PCIe bus anyway, given > >>> the tight integration with the CPU. > >>> > >>> Best, > >>> Conrad > >>> _______________________________________________ > >>> freebsd-drivers@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > >>> To unsubscribe, send any mail to " > freebsd-drivers-unsubscribe@freebsd.org > >>> " > >>> > >> > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAO%2BANN1e2Pwy1KsssQqy92_ivU_CKtO9wnd=EjmDtSLnw%2BawA>