Date: Tue, 18 Dec 2018 09:13:55 -0300 From: "O'Connor, Daniel" <darius@dons.net.au> To: Rajesh Kumar <rajfbsd@gmail.com> Cc: cem@freebsd.org, FreeBSD Hackers <freebsd-hackers@freebsd.org>, freebsd-drivers@freebsd.org, asomers@freebsd.org Subject: Re: How to use the DMA Engine in FreeBSD? Message-ID: <2DDD8ADE-4E34-4239-88EA-473B30FF8087@dons.net.au> In-Reply-To: <CAAO%2BANNHHJpMMgXuDSC7ftpTwPr1XPj9_ioYoA83WJooiELwog@mail.gmail.com> References: <CAAO%2BANOty8FeOR7VQdXmxWy5y288pm=Q4cswHJ1BbgT2h1cUWg@mail.gmail.com> <CANCZdfrEv9o6_ydp9Qe73O1v6NqHFtivHpw4m2PKhTyAVaPR9g@mail.gmail.com> <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> <CAOtMX2jXMz3D8_JvOk4M2DrEZDP8sztgy2bdBuv-yNwz9zR2JQ@mail.gmail.com> <CAG6CVpV5hL%2Bk3OcJZmMfvJnSPo5wU1w8o0A85TbBAU49a661zA@mail.gmail.com> <CAAO%2BANPa0uTvZPbq%2BM%2BX1ozA5ytUoAG6RwXhyM5dGzq1ZasFiw@mail.gmail.com> <CAAO%2BANNHHJpMMgXuDSC7ftpTwPr1XPj9_ioYoA83WJooiELwog@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 18 Dec 2018, at 05:29, Rajesh Kumar <rajfbsd@gmail.com> wrote: > Any thoughts (or) suggestions here? You don't need to "assign DMA channels" and a PCIe device doesn't = "master the bus" because it's all point to point. "DMA channels" are a legacy of ISA cards, although the terminology may = be reused for things like ioat. For PCI[e] devices which support bus mastering (ie virtually all of = them) the device itself issues the transfer. How that is actually done = depends on the device - usually you program registers with a = scatter/gather (SG) list, or with the physical address of an SG list in = system memory and then the device will do the job. The bus_dma* routines in the kernel are a tool box to let a driver tell = the kernel what kind of DMA a card is going to do and to help with = things like mapping it as necessary, cache coherency etc etc. Systems with ioat are pretty rare I believe, the vast majority of DMA in = a modern PC is done by PCI cards (or things which look like PCI cards to = the OS). > On Mon, Dec 17, 2018 at 4:54 PM Rajesh Kumar <rajfbsd@gmail.com> = wrote: >=20 >> Hi, >>=20 >> Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers = to a >> peripheral device being made as the owner of the bus by the DMA = engine and >> do data transfer. Please correct me if I am wrong. But, will this = bus_dma >> API uses the DMA engine in hardware (does it support PCI = busmastering)? >>=20 >> As I understand, DMA engine will expose DMA channels, which can be = used by >> the client drivers to queue their DMA requests (without CPU = intervention) >> and completions are given through the callback functions. If this is >> considered the real DMA operation, I don't see options to request DMA >> channels using bus_dma API's, and the callbacks in bus_dma API is = meant to >> receive just the DMA map details (doesn't indicate a DMA completion). >> ioat(4) driver seems to be doing what I described above, but that = seems to >> be Intel specific. Can it be used with any DMA controllers? >>=20 >> So, I am wondering what needs to be done in FreeBSD to do the actual = DMA >> involving DMA engine in a more generic way? >>=20 >> Thanks, >> Rajesh. >>=20 >>=20 >> On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer <cem@freebsd.org> wrote: >>=20 >>> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <asomers@freebsd.org> = wrote: >>>>> For some Intel based server hardware, there is the "ioat" driver, >>> which >>>>> allows for user code to schedule DMA operations. See ioat(4) for >>>>> details, including a pointer to the test program. >>>=20 >>> This isn't true (or at least, only for testing). It's only usable = by >>> the kernel at the moment. >>>=20 >>>> * In what context are callbacks called? Are they called from a = signal >>>> handler, or in a separate thread, or something else? >>>=20 >>> Directly from the ithread, mostly. Callers can't take sleep locks = or >>> do very much besides set a flag and wakeup, or kick off a callout or >>> taskqueue. >>>=20 >>>> * Why isn't ioat.h installed? >>>=20 >>> Why would it be? It's a kernel interface. >>>=20 >>>> * Are "interrupts" synonymous with callbacks? >>>=20 >>> I don't understand the question. Interrupts are synonymous with >>> interrupts. Like, MSI-X. >>>=20 >>>> * Do you have a rough idea for about the minimum buffer size that >>>> makes sense to use with ioat? >>>=20 >>> What makes sense will depend on your use case. It's not necessarily >>> faster than CPU memcpy, but it may be worth some slowdown to offload >>> latency-insensitive bulk copies. (It may make a lot of sense for >>> asynchronous page zero, if someone wants to bring that back.) It's >>> worth benchmarking your specific model. >>>=20 >>> We use it for >=3D 8kB copies on Broadwell, although that's largely = an >>> artifact of our use case (write sizes are exactly 8 bytes, 512 = bytes, >>> or multiples of 8kB). IIRC, it shows up as a gen3 PCIe device with >>> some small number of lanes on Broadwell, although I may be mistaken; >>> and it may not communicate with RAM over the PCIe bus anyway, given >>> the tight integration with the CPU. >>>=20 >>> Best, >>> Conrad >>> _______________________________________________ >>> freebsd-drivers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >>> To unsubscribe, send any mail to = "freebsd-drivers-unsubscribe@freebsd.org >>> " >>>=20 >>=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org" -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2DDD8ADE-4E34-4239-88EA-473B30FF8087>