Date: Mon, 17 Feb 2003 09:42:28 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Bosko Milekic <bmilekic@unixdaemons.com> Cc: Andrew Gallatin <gallatin@cs.duke.edu>, freebsd-arch@FreeBSD.ORG Subject: Re: mb_alloc cache balancer / garbage collector Message-ID: <200302171742.h1HHgSOq097182@apollo.backplane.com> References: <20030216213552.A63109@unixdaemons.com> <15952.62746.260872.18687@grasshopper.cs.duke.edu> <20030217095842.D64558@unixdaemons.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The work looks great, but I have to say I have grave reservations
over any sort of thread-based cleanup / balancing code for a memory
allocation subsystem. The only advantage that I can see is that you
get good L1 cache effects, but that is counterbalanced by a number of
severe disadvantages which have taken a long time to clear up in other
subsystems which use separate threads (pageout daemon, buf daemon,
syncer, pmap garbage collection from the pageout daemon, etc). Most of
these daemons have very good reasons for needing a thread, but I can't
think of any reason why a straight memory allocator would *require*
a thread.
Wouldn't it be easier and more scaleable to implement the hysteresis on
the fly? It sounds like it ought to be simple... you have a sysctl
to set the per-cpu free cache size and hysteresis (for example, 32[8],
aka upon reaching 32 free 32 - 8 = 24 to the global cache, keeping 8).
Overflow goes into a global pool. Active systems do not usually
bounce from 0 to the maximum number of mbufs and back again, over
and over again. Instead they tend to have smaller swings and 'drift'
towards the edges, so per-cpu hysteresis should not have to exceed
10% of the total available buffer space in order to reap the maximum
locality of reference and mutex benefit. Even in a very heavily loaded
system I would expect something like 128[64] to be sufficient. This
sort of hysteresis could be implemented trivially in the main mbuf
freeing code without any need for a thread and would have the same
performance / L1 cache characteristics. Additionally, on-the-fly
hysteresis would be able to handle extreme situations that a thread
could not (such as extreme swings), and on-the-fly hysteresis can
scale in severe or extreme situations while a thread cannot.
The same argument could also be applied to UMA, btw.
-Matt
Matthew Dillon
<dillon@backplane.com>
:
:
:On Mon, Feb 17, 2003 at 09:43:38AM -0500, Andrew Gallatin wrote:
:>
:> Bosko Milekic writes:
:> > What does this mean for us in the really short term? It means that we
:> > can finally make all M_DONTWAIT allocations NOT interface with the VM
:> > subsystem at all. Why is this good in the really really short term?
:> > For one, you can have network device drivers call the mbuf code
:> > without Giant because they'll know for a fact that Giant will never be
:> > needed down the line. Since the cache balancer will replenish caches
:>
:> Not to detract from your work, but M_DONTWAIT allocations have been
:> Giant-free since Alan made the VM system Giant-free for kernel-map
:> allocations in early January.
:>
:> The long-term implications of this work look very exciting.
:
: Yep, just noticed that. Oh well, all the better for us! I haven't
: actually instrumented M_DONTWAIT allocations to NOT dip into the VM
: code in the patch so I guess I won't have to after all.
:
:> Drew
:
:--
:Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200302171742.h1HHgSOq097182>