Date: Mon, 25 Nov 2002 16:20:39 -0500 (EST) From: Andrew Gallatin <gallatin@cs.duke.edu> To: Bosko Milekic <bmilekic@unixdaemons.com> Cc: Julian Elischer <julian@elischer.org>, Robert Watson <rwatson@freebsd.org>, Luigi Rizzo <rizzo@icir.org>, current@freebsd.org Subject: Re: mbuf header bloat ? Message-ID: <15842.37927.919140.910293@grasshopper.cs.duke.edu> In-Reply-To: <20021125160122.A75673@unixdaemons.com> References: <15840.8629.324788.887872@grasshopper.cs.duke.edu> <Pine.BSF.4.21.0211232306410.28833-100000@InterJet.elischer.org> <15841.17237.826666.653505@grasshopper.cs.duke.edu> <20021125130005.A75177@unixdaemons.com> <15842.27547.385354.151541@grasshopper.cs.duke.edu> <20021125160122.A75673@unixdaemons.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Bosko Milekic writes:
> Firstly, it should be noted that the behavior of calling kmem_malloc()
> when its caches are empty is an old property that has been carried
> over from the original allocator - in other words, it is not something
> that I arbitrarily introduced.
Certainly. I'm not blaming you for that.
> With that said, we may want to think about changing it or at least
> introducing a third case that would modify the semantics so that Giant
> would never be called. For instance, in addition to M_TRYWAIT and
> M_DONTWAIT, you could invent M_CACHEONLY that would only attempt a
> quick cache allocation. However, I'm unsure as to the advantages this
> would bring - if any. Why is it so much of a problem if your driver
> wants to grab Giant? Are you afraid of lock order reversals somewhere
> down the line? Perhaps there is a misconception that we can clear up
> here before any work is done.
>
I'm not sure what you mean. The problem is that Giant inhibits
scaling on SMPs and I want to get all network drivers out from under
it, not just mine.
Requiring Giant for mbuf allocations effectivly defeats your elegant,
cleverly designed mbuf allocator with lock-free per-cpu caches, etc.
And it introduces all kinds of windows for races and locking errors if
SMP safe drivers must drop all their mutexes and grab Giant for mbuf
allocations.
And you cannot grab Giant inside the mbuf allocation code itself
because the mutex rules prohibit acquiring Giant while holding any
other locks. From mutex(9):
Giant
If Giant must be acquired, it must be acquired prior to acquiring other
mutexes. Put another way: it is impossible to acquire Giant non-recur-
sively while holding another mutex. It is possible to acquire other
mutexes while holding Giant, and it is possible to acquire Giant recur-
sively while holding other mutexes.
Emperically, I vaguely remember marking my driver as SMP safe (with
witness and invariants off, of course) provided something like a
30-40% performance increase on a dual 1GHz PIII system. It was still
not as fast as stable, but the current/stable performance difference
was no longer embarrassing.
Drew
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?15842.37927.919140.910293>