Date: Mon, 25 Nov 2002 13:00:05 -0500 From: Bosko Milekic <bmilekic@unixdaemons.com> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: Julian Elischer <julian@elischer.org>, Robert Watson <rwatson@freebsd.org>, Luigi Rizzo <rizzo@icir.org>, current@freebsd.org Subject: Re: mbuf header bloat ? Message-ID: <20021125130005.A75177@unixdaemons.com> In-Reply-To: <15841.17237.826666.653505@grasshopper.cs.duke.edu>; from gallatin@cs.duke.edu on Sun, Nov 24, 2002 at 04:23:33PM -0500 References: <15840.8629.324788.887872@grasshopper.cs.duke.edu> <Pine.BSF.4.21.0211232306410.28833-100000@InterJet.elischer.org> <15841.17237.826666.653505@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Nov 24, 2002 at 04:23:33PM -0500, Andrew Gallatin wrote: > If we're going to nitpick the mbuf system, a much, much worse problem > is that you cannot allocate an mbuf chain w/o holding Giant, which > stems from the mbuf system eventually calling kmem_malloc(). This > effectively prevents any network driver from being giant-free. When > mbufs are low, mb_alloc() calls mb_pop_cont(). This, in turn, calls > kmem_malloc() which requires Giant... This is not entirely true. You can allocate an mbuf chain without holding Giant if the caches are well populated - and they should be in the common/general case. You can in fact modify the allocator to just not do a kmem_malloc() if called with M_DONTWAIT, but I don't think you'd want to do this at this point. > The mbuf system calls malloc in other ways too. The first time you > use a cluster, m_ext.ext_ref_cnt is malloc()'ed, and malloc is called > when the mbuf map is expanded... I assume malloc will eventually > call kmem_malloc(), leading to the same locking problems. Actually, that has been changed a while ago. The ref_cnt for clusters is no longer malloc()'d. A contiguous space is used from which cluster ref counters are "allocated" (just like in the old/original design). This modification was made a while ago as an optimisation. > I know that both tru64 and aix just malloc their mbufs. The main reason for which mbufs and clusters are allocated via a different allocator boils down to significant optimisations. Some of these include grouped mbuf + cluster allocations with minimized lock-dropping/re-acquiring for grouped allocations (essentially "atomic allocations"), reduced lock contention, not being forced to call VM routines in the common free case (i.e., maybe optionally implement lazy freeing), reducing the number of function calls required to make a single buffer allocation, etc, etc, etc. The gist of the argument boils down to the fact that network buffer allocations have different requirements than general all-purpose allocations (by design, the last time I checked), and that is why an mbuf/cluster allocator exists. With that said, the mbuf/cluster allocator shares some important characteristics with the UMA implementation of malloc (if I remember UMA the last time I checked) and those have to do with, primarily, SMP-friendliness. > Drew -- Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021125130005.A75177>