Date: Tue, 31 Oct 2000 11:39:21 +0100 From: Andre Oppermann <oppermann@telehouse.ch> To: Nat Lanza <magus@cs.cmu.edu> Cc: Alfred Perlstein <bright@wintelcom.net>, Bosko Milekic <bmilekic@dsuper.net>, freebsd-net@FreeBSD.ORG Subject: Re: MP: per-CPU mbuf allocation lists Message-ID: <39FEA159.EB774F1E@telehouse.ch> References: <Pine.BSF.4.21.0010301256580.30271-100000@jehovah.technokratis.com> <20001030104457.E22110@fw.wintelcom.net> <39FDD039.594CED43@telehouse.ch> <uoc7l6pyail.fsf@hurlame.pdl.cs.cmu.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Nat Lanza wrote:
>
> Andre Oppermann <oppermann@telehouse.ch> writes:
>
> > Lets assume there is an apache webserver running on the MP machine.
> > Now this apache process tends to stick to one CPU (does it? would
> > make sense for cache locality). This process generates a ton of
> > traffic and needs thousands of mbufs. If I keep the watermark
> > defaults then this is sub-optimal because on CPU1 I need thousands
> > of mbuf whereas CPU2 is not using much. If I tune the watermarks to
> > the required level of CPU1 then I'm wasting an awful lot of mbufs on
> > the other CPU('s)...
>
> Actually, Apache uses a pool of processes. So ideally the active httpd
> processes would be distributed fairly evenly across the machine's CPUs.
Depends, if you look at Apache 2.0 it will run multithreaded. In the
Apache case it could also make sense to keep the childs on the same
CPU to avoid cache line thrashing. There are a lot of informations
that flow between the parent Apache and the Childs (requests, filters,
log entries...). It makes sense to run cgi's on a different CPU.
> Also, if there is only one process heavily consuming mbufs, then
> presumably acquiring the locks necessary to grab mbufs from the global
> pool is fast, since there's not going to be any contention for them.
No. First, you contradict yourself here and second if two Apache childs
are simultaniusly running on two CPU's they are competing at the same
time for the resource mbuf and one will starve the other and we have a
lot of contention.
> > The proposal; Lets have the minimum watermark set on a system-wide
> > level (sysctl) but the high-watermark would be based per-cpu with
> > an sliding window calculating the average use of mbufs over a certain
> > period of time (eg. 10sec). The initial high watermark would be
> > either min*2 or set upon system initialization to some initial (base)
> > value. This also takes automagically care of process migration between
> > the CPU's. Also this system would be self adjusting and self tuning
> > without admin intervention in common edge cases.
>
> I'm not convinced this is really necessary; if there isn't any
> contention for the mbuf lists, then it doesn't matter too much where
> they are -- as I understand it, the benefit in per-cpu mbuf lists is
> that you avoid contention when there are multiple consumers of
> mbufs. If you don't have multiple consumers, you're not going to have
> contention, so I don't see the point in adding fancy logic for that
> case.
You have multiple mbuf consumers, see above. Also I made a point about
wasting mbufs when you tune the high watermark for high loads.
--
Andre
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?39FEA159.EB774F1E>