Date: Wed, 8 Nov 2000 13:13:43 -0600 From: Jonathan Lemon <jlemon@flugsvamp.com> To: smp@freebsd.org Subject: SMP safe interface queues Message-ID: <20001108131343.A72943@prism.flugsvamp.com>
index | next in thread | raw e-mail
I've converted the ifqueue structure and the IF* macros to be SMP
safe. The conversion was done with an eye towards modifying the
ethernet drivers as little as possible. An explanation of the
changes is below, and at the end is a design question of what to
do with "misbehaved" drivers.
Borrowing from the mbuf model, the existing IF_DEQUEUE, IF_ENQUEUE
and IF_PREPEND macros are prepended with an underscore, and new
macros are defined that obtain and release the lock around the
call. E.g.:
#define IF_DEQUEUE(ifq, m) do { \
IF_LOCK(ifq); \
_IF_DEQUEUE(ifq, m); \
IF_UNLOCK(ifq); \
} while (0)
#define IF_PREPEND(ifq, m) do { \
IF_LOCK(ifq); \
_IF_PREPEND(ifq, m); \
IF_UNLOCK(ifq); \
} while (0)
With this approach, no change at all is required for most drivers.
Mutex initialization/destruction is done by if_attach/if_detach().
Most of the upper layer uses a variant of the following model:
s = splnet();
if (IF_QFULL(ifq)) {
IF_DROP(ifq);
m_freem(m);
return;
}
ifp->if_obytes += m->m_pkthdr.len;
if (m->m_flags & M_MCAST)
ifp->if_omcasts++;
IF_ENQUEUE(ifq, m);
if ((ifp->if_flags & IFF_OACTIVE) == 0)
(*ifp->if_start)(ifp);
splx(s);
When converting this to use mutexes, the lock has to be dropped before
calling the start routine, otherwise we can deadlock. This opens up
a possible race where the driver can turn off the OACTIVE flag after
a packet is queued. So to make things simpler, the entire block of
code above is moved into a new "IF_HANDOFF" macro which takes care of
all the details for the caller. The upper layer code simplifies to:
if (! IF_HANDOFF(ifq, m, ifp))
ifp->if_oerrors++;
For full details, refer to the current patchset against -current which
is at: http://www.flugsvamp.com/~jlemon/fbsd/ifq_v3.patch
The only warts in this particular model are certain drivers which
abuse the ifqueue structure as a convenient way of retaining mbufs
that the chip is transmitting. In this case, the ifq is completely
private to one driver, and thus does not need to be locked, as it
is not an interface queue at all. The question then is should the
driver be converted to use the "mutex-free" versions of the macros
above, or should it be forced to perform locking/unlocking even though
it is unncessary?
Some example drivers doing this are midway, de, awi, hea, lmc.
Personally, I'd prefer to have these use the 'mutex-free' version of
the macro to make it clear that no locks are being used, but this may
be a problem for maintainability, especially considering that most
(if not all) of the above drivers are from NetBSD.
Feedback welcome.
--
Jonathan
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20001108131343.A72943>