From owner-freebsd-net Tue Oct 31 2:38:47 2000 Delivered-To: freebsd-net@freebsd.org Received: from mailtoaster1.pipeline.ch (mailtoaster1.pipeline.ch [62.48.0.70]) by hub.freebsd.org (Postfix) with SMTP id 22A2637B4C5 for ; Tue, 31 Oct 2000 02:38:44 -0800 (PST) Received: (qmail 66745 invoked from network); 31 Oct 2000 10:37:19 -0000 Received: from unknown (HELO telehouse.ch) ([62.48.0.53]) (envelope-sender ) by mailtoaster1.pipeline.ch (qmail-ldap-1.03) with SMTP for ; 31 Oct 2000 10:37:19 -0000 Message-ID: <39FEA159.EB774F1E@telehouse.ch> Date: Tue, 31 Oct 2000 11:39:21 +0100 From: Andre Oppermann X-Mailer: Mozilla 4.74 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Nat Lanza Cc: Alfred Perlstein , Bosko Milekic , freebsd-net@FreeBSD.ORG Subject: Re: MP: per-CPU mbuf allocation lists References: <20001030104457.E22110@fw.wintelcom.net> <39FDD039.594CED43@telehouse.ch> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Nat Lanza wrote: > > Andre Oppermann writes: > > > Lets assume there is an apache webserver running on the MP machine. > > Now this apache process tends to stick to one CPU (does it? would > > make sense for cache locality). This process generates a ton of > > traffic and needs thousands of mbufs. If I keep the watermark > > defaults then this is sub-optimal because on CPU1 I need thousands > > of mbuf whereas CPU2 is not using much. If I tune the watermarks to > > the required level of CPU1 then I'm wasting an awful lot of mbufs on > > the other CPU('s)... > > Actually, Apache uses a pool of processes. So ideally the active httpd > processes would be distributed fairly evenly across the machine's CPUs. Depends, if you look at Apache 2.0 it will run multithreaded. In the Apache case it could also make sense to keep the childs on the same CPU to avoid cache line thrashing. There are a lot of informations that flow between the parent Apache and the Childs (requests, filters, log entries...). It makes sense to run cgi's on a different CPU. > Also, if there is only one process heavily consuming mbufs, then > presumably acquiring the locks necessary to grab mbufs from the global > pool is fast, since there's not going to be any contention for them. No. First, you contradict yourself here and second if two Apache childs are simultaniusly running on two CPU's they are competing at the same time for the resource mbuf and one will starve the other and we have a lot of contention. > > The proposal; Lets have the minimum watermark set on a system-wide > > level (sysctl) but the high-watermark would be based per-cpu with > > an sliding window calculating the average use of mbufs over a certain > > period of time (eg. 10sec). The initial high watermark would be > > either min*2 or set upon system initialization to some initial (base) > > value. This also takes automagically care of process migration between > > the CPU's. Also this system would be self adjusting and self tuning > > without admin intervention in common edge cases. > > I'm not convinced this is really necessary; if there isn't any > contention for the mbuf lists, then it doesn't matter too much where > they are -- as I understand it, the benefit in per-cpu mbuf lists is > that you avoid contention when there are multiple consumers of > mbufs. If you don't have multiple consumers, you're not going to have > contention, so I don't see the point in adding fancy logic for that > case. You have multiple mbuf consumers, see above. Also I made a point about wasting mbufs when you tune the high watermark for high loads. -- Andre To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message