From owner-freebsd-arch Mon Feb 17 19:45:53 2003 Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9AED37B401 for ; Mon, 17 Feb 2003 19:45:50 -0800 (PST) Received: from sccrmhc02.attbi.com (sccrmhc02.attbi.com [204.127.202.62]) by mx1.FreeBSD.org (Postfix) with ESMTP id 38F9643F3F for ; Mon, 17 Feb 2003 19:45:49 -0800 (PST) (envelope-from julian@elischer.org) Received: from interjet.elischer.org (12-232-168-4.client.attbi.com[12.232.168.4]) by sccrmhc02.attbi.com (sccrmhc02) with ESMTP id <200302180345480020032booe>; Tue, 18 Feb 2003 03:45:48 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id TAA18135; Mon, 17 Feb 2003 19:45:47 -0800 (PST) Date: Mon, 17 Feb 2003 19:45:45 -0800 (PST) From: Julian Elischer To: Bosko Milekic Cc: freebsd-arch@FreeBSD.ORG Subject: Re: mb_alloc cache balancer / garbage collector In-Reply-To: <20030217221349.A67942@unixdaemons.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Mon, 17 Feb 2003, Bosko Milekic wrote: > > On Mon, Feb 17, 2003 at 05:38:27PM -0800, Julian Elischer wrote: > > On Mon, 17 Feb 2003, Bosko Milekic wrote: > > [...] > > Bosko, If I have one NIC bound to one CPU (a future capability say,) > > and another bound to a second, and there is a stream of packets fron > > NIC1 to NIC2 (we are routing) at (say) 30,000 packets per second, > > what is the path by which those 30,000 packets make their way from > > CPU2's cache of mbufs, back to CPU1 to be used again? (each second). > > We'll imagine there are no packets going the other way. (maybe they take > > a different route). > > If the mbufs are allocated from CPU1's cache then they'll most likely > be freed back to CPU1's cache. The way it works is that the mbuf is > freed back to its bucket and so to whichever cache the bucket is > sitting in. Most likely the bucket will not have migrated caches so > you're going to be using CPU1's cache in this scenario, since that's > the cache you're allocating from. This is probably not ideal when you > do something like bind the NIC to a CPU but it is better than having > the freing thread free to its own cache, in which case you'd have a > serious debalancing of caches going on. I'm not sure how performance > would be impacted either way, but I guess I can't say until we > actually bind the NICs each to their own CPUs and measure. So this means that CPU2 is freeing into a cache belonging to CPU1. This means that somewhere there must be a lock involved.. I thought that part of this was that we were trying to avoid using locks.. Is there a separation between the structures that accept the buffer from CPU2 and those that CPU1 gets them from? What mitigates the lock contention between CPU2 and CPU1? > > -- > Bosko Milekic * bmilekic@unixdaemons.com * bmilekic@FreeBSD.org > > "If we open a quarrel between the past and the present, we shall > find that we have lost the future." > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message