Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 11 Jun 2010 11:21:24 -0600
From:      Scott Long <scottl@samsco.org>
To:        Marcel Moolenaar <xcllnt@mac.com>
Cc:        svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, John Baldwin <jhb@freebsd.org>
Subject:   Re: svn commit: r209026 - in head/sys/ia64: ia64 include
Message-ID:  <EE4B5C07-194D-4408-B0A1-BA25BE39468F@samsco.org>
In-Reply-To: <19B0DF11-5998-40F5-8095-8D2521B1C597@mac.com>
References:  <201006110300.o5B30X9q045387@svn.freebsd.org> <201006110751.40735.jhb@freebsd.org> <853068F6-D736-4DA3-859F-D946D096843D@samsco.org> <19B0DF11-5998-40F5-8095-8D2521B1C597@mac.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Jun 11, 2010, at 11:04 AM, Marcel Moolenaar wrote:
>=20
> On Jun 11, 2010, at 9:12 AM, Scott Long wrote:
>=20
>> On Jun 11, 2010, at 5:51 AM, John Baldwin wrote:
>>> On Thursday 10 June 2010 11:00:33 pm Marcel Moolenaar wrote:
>>>> Author: marcel
>>>> Date: Fri Jun 11 03:00:32 2010
>>>> New Revision: 209026
>>>> URL: http://svn.freebsd.org/changeset/base/209026
>>>>=20
>>>> Log:
>>>> Bump MAX_BPAGES from 256 to 1024. It seems that a few drivers, =
bge(4)
>>>> in particular, do not handle deferred DMA map load operations at =
all.
>>>> Any error, and especially EINPROGRESS, is treated as a hard error =
and
>>>> typically abort the current operation. The fact that the busdma =
code
>>>> queues the load operation for when resources (i.e. bounce buffers =
in
>>>> this particular case) are available makes this especially =
problematic.
>>>> Bounce buffering, unlike what the PR synopsis would suggest, works
>>>> fine.
>>>>=20
>>>> While on the subject, properly implement swi_vm().
>>>=20
>>> NIC drivers do not handle deferred load operations at all (note that=20=

>>> bus_dmamap_load_mbuf() and bus_dmamap_load_mbuf_sg() enforce =
BUS_DMA_NOWAIT).
>>> It is common practice to just drop the packet in that case.
>>>=20
>>=20
>> Yes, long ago when network drivers started being converted to busdma, =
it was agreed that EINPROGRESS simply doesn't make sense for them.  Any =
platform that winds up making extensive use of bounce buffers for =
network hardware is going to perform poorly no matter what, and should =
hopefully have some sort of IOMMU that can be used instead.
>=20
> Unfortunately things aren't as simple as is presented.
>=20
> For one, bge(4) wedges as soon as the platform runs out of bounce
> buffers when they're needed. The box needs to be reset in order to
> get the interface back. I pick any implementation that remains
> functional over a mis-optimized one that breaks. Deferred load
> operations are more performance optimal than failure is.
>=20

This sounds like a bug in the bge driver.  I don't see if through casual =
inspection, but the driver should be able to either drop the mbuf =
entirely, or requeue it on the ifq and then restart the ifq later.

> Also: the kernel does nothing to guarantee maximum availability
> of DMA-able memory under load, so bounce buffers (or use of I/O
> MMUs for that matter) are a reality. Here too the performance
> argument doesn't necessarily hold because the kernel may be
> busy with more than just sending and receiving packets and the
> need to defer load operations is very appropriate. If the
> alternative is just dropped packets, I'm fine with that too,
> but I for one cannot say that *not* filling a H/W ring with
> buffers is not going to wedge the hardware in some cases.
>=20
> Plus: SGI Altix does not have any DMA-able memory for 32-bit
> hardware. The need for an I/O MMU is absolute and since there
> are typically less mapping registers than packets on a ring,
> the need for deferred operation seems quite acceptable if the
> alternative is, again, failure to operate.
>=20

I'm not against you upping the bounce buffer limit for a particular =
platform, but it's still unclear to me if (given bug-free drivers) it's =
worth the effort to defer a load rather than just drop the packet and =
let the stack retry it.  One question that would be good to answer is =
wether the failed load is happening in the RX to TX path.

Scott




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EE4B5C07-194D-4408-B0A1-BA25BE39468F>