Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 14 Dec 2018 16:59:37 -0800
From:      Conrad Meyer <cem@freebsd.org>
To:        Alan Somers <asomers@freebsd.org>
Cc:        "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, freebsd-drivers@freebsd.org
Subject:   Re: How to use the DMA Engine in FreeBSD?
Message-ID:  <CAG6CVpV5hL%2Bk3OcJZmMfvJnSPo5wU1w8o0A85TbBAU49a661zA@mail.gmail.com>
In-Reply-To: <CAOtMX2jXMz3D8_JvOk4M2DrEZDP8sztgy2bdBuv-yNwz9zR2JQ@mail.gmail.com>
References:  <CAAO%2BANOty8FeOR7VQdXmxWy5y288pm=Q4cswHJ1BbgT2h1cUWg@mail.gmail.com> <CANCZdfrEv9o6_ydp9Qe73O1v6NqHFtivHpw4m2PKhTyAVaPR9g@mail.gmail.com> <26df913b-a2f8-2709-1cec-d11ad7d113a8@pix.net> <CAOtMX2jXMz3D8_JvOk4M2DrEZDP8sztgy2bdBuv-yNwz9zR2JQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <asomers@freebsd.org> wrote:
> > For some Intel based server hardware, there is the "ioat" driver, which
> > allows for user code to schedule DMA operations.  See ioat(4) for
> > details, including a pointer to the test program.

This isn't true (or at least, only for testing).  It's only usable by
the kernel at the moment.

> * In what context are callbacks called?  Are they called from a signal
> handler, or in a separate thread, or something else?

Directly from the ithread, mostly.  Callers can't take sleep locks or
do very much besides set a flag and wakeup, or kick off a callout or
taskqueue.

> * Why isn't ioat.h installed?

Why would it be?  It's a kernel interface.

> * Are "interrupts" synonymous with callbacks?

I don't understand the question.  Interrupts are synonymous with
interrupts.  Like, MSI-X.

> * Do you have a rough idea for about the minimum buffer size that
> makes sense to use with ioat?

What makes sense will depend on your use case.  It's not necessarily
faster than CPU memcpy, but it may be worth some slowdown to offload
latency-insensitive bulk copies.  (It may make a lot of sense for
asynchronous page zero, if someone wants to bring that back.)  It's
worth benchmarking your specific model.

We use it for >= 8kB copies on Broadwell, although that's largely an
artifact of our use case (write sizes are exactly 8 bytes, 512 bytes,
or multiples of 8kB).  IIRC, it shows up as a gen3 PCIe device with
some small number of lanes on Broadwell, although I may be mistaken;
and it may not communicate with RAM over the PCIe bus anyway, given
the tight integration with the CPU.

Best,
Conrad



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG6CVpV5hL%2Bk3OcJZmMfvJnSPo5wU1w8o0A85TbBAU49a661zA>