Date: Fri, 07 Jul 2000 09:16:20 -0600 From: Chuck Paterson <cp@berserker.bsdi.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: Doug Rabson <dfr@nlsystems.com>, Greg Lehey <grog@lemis.com>, David Greenman <dg@root.com>, freebsd-smp@freebsd.org Message-ID: <200007071516.JAA12975@berserker.bsdi.com>
next in thread | raw e-mail | index | archive | help
Subject: Spin Locks, blocking interrupts, and ipending
From: Chuck Paterson <cp@bsdi.com>
Fcc: outbound
--------
Matthew Dillon wrote on: Thu, 06 Jul 2000 10:43:23 PDT
}
} There are two ways to do this:
}
} (1) (The way I implemented it)
Removed stuff
} Disadvantages:
}
} None.
}
One disadvantage is that the actual mechanism to implement
the lazy interrupts is going to be a bitch to actually make work
right/well on multiple processors in parallel, further complicated
by the light weight context switches. Also I'm pretty sure interrupts
don't want to get pended to by the pic level, but rather on individual
apic pins.
A second disadvantage is that interrupts will get randomly delivered to
a processor holding a spin mutex rather than run immediately on a
processor not holding a spin mutex. This is a problem with the
current scheme also, and not fixed with masks and ipending.
Having said this, it may be that Matt is correct and in the long
term we will want to ditch the cli/sti's in mutices. I really hope
we don't have to go down the lazy path, my mind goes numb just
thinking about the corner cases.
Some random data points.
The assumption that the scheduler lock will be the only
spin mutex is wrong. BSD/OS currently has about 5 which
are pretty much architecture independent. Most/all of these
are at one time or another acquired while the scheduler
lock is held.
On Sparc BSD/OS already has the notion of a priority
level associated with a spin lock. Interrupts are blocked
by changing the interrupting block level, which on Sparc
is very cheap.
When a processor is cli'd IPIs are not delivered. In general
this made life lots easier in BSD/OS. It is possible that
a condition will arise where is it required to deliver
an IPI while another processor holds a spin lock. This
did occur on Sparc with some low cache code (stuff done
in hardware on Intel). This particular IPI is delivered
at a level higher than just acquiring a mutex will not block.
It is likely that some devices will want a lower level half
that always runs in a borrowed context and is protected
by spin locks. Currently in BSD/OS the Sparc zs driver
falls into this category. It sure seems likely that
the X86 com driver will also fall into this category. Using
sti/cli does not preclude this, but it does preclude
a driver of this type interrupting another driver of this
type.
In order to avoid a deadly embrace spin mutice must block all
interrupts of mutice which which are logically before them in
the locking order. This lends itself to a priority scheme for
blocking rather than a unordered bit mask. This is not to
say that a bit mask couldn't be used to implement a priority
scheme.
The task priority register could used to block interrupts on
Pentium. The code could then work much like Sparc. This has
the advantage of allowing some spin mutice to be interrupted
without the added complexity of ipending. The real good thing
is that the interrupt will get dispatched to a processor able
to handle it, rather than just being pended on a processor
which has it blocked. I don't yet know how expensive writes to
the task priority register are.
The cost of spin mutice are not that big of a deal. They are the
more expensive mutex, and generally when the are acquired
something very expensive, like a task switch, is already
occuring. The cost for use with drivers like the com driver is
not an issue. Just taking the interrupt and then talking to the
hardware totally swamps the cost of the mutex.
Chuck
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200007071516.JAA12975>