Date: Mon, 17 Feb 2003 23:03:27 -0500 From: Bosko Milekic <bmilekic@unixdaemons.com> To: Julian Elischer <julian@elischer.org> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: mb_alloc cache balancer / garbage collector Message-ID: <20030217230327.A68207@unixdaemons.com> In-Reply-To: <Pine.BSF.4.21.0302171940420.16142-100000@InterJet.elischer.org>; from julian@elischer.org on Mon, Feb 17, 2003 at 07:45:45PM -0800 References: <20030217221349.A67942@unixdaemons.com> <Pine.BSF.4.21.0302171940420.16142-100000@InterJet.elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 17, 2003 at 07:45:45PM -0800, Julian Elischer wrote: > On Mon, 17 Feb 2003, Bosko Milekic wrote: > > On Mon, Feb 17, 2003 at 05:38:27PM -0800, Julian Elischer wrote: > > > On Mon, 17 Feb 2003, Bosko Milekic wrote: > > > [...] > > > Bosko, If I have one NIC bound to one CPU (a future capability say,) > > > and another bound to a second, and there is a stream of packets fron > > > NIC1 to NIC2 (we are routing) at (say) 30,000 packets per second, > > > what is the path by which those 30,000 packets make their way from > > > CPU2's cache of mbufs, back to CPU1 to be used again? (each second). > > > We'll imagine there are no packets going the other way. (maybe they take > > > a different route). > > > > If the mbufs are allocated from CPU1's cache then they'll most likely > > be freed back to CPU1's cache. The way it works is that the mbuf is > > freed back to its bucket and so to whichever cache the bucket is > > sitting in. Most likely the bucket will not have migrated caches so > > you're going to be using CPU1's cache in this scenario, since that's > > the cache you're allocating from. This is probably not ideal when you > > do something like bind the NIC to a CPU but it is better than having > > the freing thread free to its own cache, in which case you'd have a > > serious debalancing of caches going on. I'm not sure how performance > > would be impacted either way, but I guess I can't say until we > > actually bind the NICs each to their own CPUs and measure. > > So this means that CPU2 is freeing into a cache belonging to CPU1. > This means that somewhere there must be a lock involved.. I thought that > part of this was that we were trying to avoid using locks.. > Is there a separation between the structures that accept the buffer from > CPU2 and those that CPU1 gets them from? What mitigates the lock > contention between CPU2 and CPU1? Right, it basically means that in this scenario we degenerate to a single cache. The structure to which the mbuf is freed is called a "bucket" and right now a "bucket" keeps a PAGE_SIZE worth of mbufs. The idea is that you can move these buckets around from cache to cache, even if they're not totally full. In the scenario that you describe (which by the way is still inexistent), assuming that we determine that it's really worth doing the binding of the threads to individual CPUs (I'm not quite convinced that it is, ... yet), in order to reduce lock contention it would probably be wise to migrate buckets over to the second CPU. Unfortunately, this is easier said than done... once you allow objects allocated from one cache to be freed to whatever cache the CPU currently tampering with them/consuming them owns, then you risk severely deblancing your caches for regular non-CPU-bound applications. So for instance you could have something like this happen: CPU 1 exhausting its cache CPU 2 blowing up its cache because freeing is suddenly done to buckets which are now being migrated to CPU2's cache. CPU 2 migrating buckets to the global cache because suddenly its cache has more than high watermark mbufs. CPU 1 taking buckets back from the global cache because its cache is exhausted. Again, this is OK if you assume that the only users of the allocator are those CPU-bound-threads. Once you start introducing other non-CPU bound threads into the picture you may find that it's better to just keep the current behavior. This is why I can't say for sure how performance will fair in either case. We really have to wait until we're ready, then implement it, then tune and measure. When designing and implementing this you can't get the best case to consistently occur for all scenarios. If we had one fixed model/way to do things then we could easily tune to make the "best case" fit that model but there are always going to be tradeoffs. -- Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org "If we open a quarrel between the past and the present, we shall find that we have lost the future." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030217230327.A68207>