Date: Thu, 08 Dec 2011 13:25:25 -0500 From: John Baldwin <jhb@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: svn-src-head@freebsd.org, Luigi Rizzo <luigi@freebsd.org>, src-committers@freebsd.org, svn-src-all@freebsd.org, Marius Strobl <marius@alchemy.franken.de> Subject: Re: svn commit: r228281 - in head/sys/dev: e1000 re Message-ID: <4EE10115.3050309@FreeBSD.org> In-Reply-To: <20111208182932.GA82488@onelab2.iet.unipi.it> References: <201112051533.pB5FXEuh043063@svn.freebsd.org> <20111205183854.GC54475@alchemy.franken.de> <20111205193150.GB49118@onelab2.iet.unipi.it> <4EE0F89E.1070107@FreeBSD.org> <20111208182932.GA82488@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/8/11 1:29 PM, Luigi Rizzo wrote:
> On Thu, Dec 08, 2011 at 12:49:18PM -0500, John Baldwin wrote:
>> On 12/5/11 2:31 PM, Luigi Rizzo wrote:
>>> On Mon, Dec 05, 2011 at 07:38:54PM +0100, Marius Strobl wrote:
>>>> On Mon, Dec 05, 2011 at 03:33:14PM +0000, Luigi Rizzo wrote:
>>> ...
>>>>> +#ifdef DEV_NETMAP
>>>>> + if (slot) {
>>>>> + int si = i + na->tx_rings[txr->me].nkr_hwofs;
>>>>> + void *addr;
>>>>> +
>>>>> + if (si>= na->num_tx_desc)
>>>>> + si -= na->num_tx_desc;
>>>>> + addr = NMB(slot + si);
>>>>> + txr->tx_base[i].buffer_addr =
>>>>> + htole64(vtophys(addr));
>>>>> + /* reload the map for netmap mode */
>>>>> + netmap_load_map(txr->txtag,
>>>>> + txbuf->map, addr, na->buff_size);
>>>>> + }
>>>>> +#endif /* DEV_NETMAP */
>>>>
>>>> Can these vtophys(9) usages be fixed to use bus_dma(9) instead so netmap
>>>> works with bounce buffers, IOMMUs etc?
>>>
>>> maybe. Can you suggest how to change it ?
>>>
>>> Consider that (not here but in other places) vtophys() is called
>>> in a time-critical loop so performance matters a lot. As long as i
>>> can compute the physical address in advance and cache it in my own
>>> array, i suppose that should be fine (in which case the calls to
>>> vtophys(addr) would become NMPB(slot + si) where the NMPB() macro
>>> would hide translations and checks.
>>
>> For your use case, you probably don't want to be coping with bounce
>> buffers at all. That is, if you are preallocating long-lived buffers
>> that keep getting reused while netmap is active that are allocated at
>> startup and free'd at teardown, you probably want to allocate buffers
>> that won't require bounce buffers. That means you have to let the
>> drivers allocate the buffers (or give you a suitable bus_dma tag since
>> different devices have different addressing requirements, etc.). You
>> could then use bus_dmamem_alloc() to allocate your buffers.
>
> certainly i don't want to use netmap with bounce buffers.
> I am not sure about IOMMU (I basically don't need it but maybe
> using a compatible API is always nice). Right now i am allocating
> a huge chunk of memory with contigmalloc.
> Ryan Stone suggested that a plain malloc may work as well (as long
> as i make sure that each buffer is within a single page).
> Eventually I may want to play with cache alignment (also suggested by Ryan)
> so allocate smaller chunks of contigmalloc'ed memory (say each buffer
> is 2K - 64 bytes, then a contiguous block of 64K fits exactly 33 buffers).
Right, you should use bus_dmamem_alloc() instead. Internally it calls
contigmalloc(). However, it will only allocate memory that your device
can safely use (so if you are using a NIC that can only do 32-bit DMA
addresses, it will allocate pages below 4GB avoiding bounce buffering).
For the IOMMU case it will usually allocate memory from the region
that is statically mapped into the IOMMU (at least some IOMMU's split
the DVMA address space into two types: a mostly static region for things
like descriptor rings, etc. and a dynamic region for transient I/O
buffers like mbufs). If you want to design an interface
that will work with a wide variety of hardware and not just ixgbe on
Intel, then using bus_dma to manage DMA buffers is the correct approach.
Also, back to IOMMU, if the device is doing DMA into this buffer, then
you _must_ use the IOMMU on certain platforms (e.g. sparc64, probably
some embedded platforms where netmap might be very nice to have).
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EE10115.3050309>