From owner-freebsd-smp Sun Nov 2 10:45:56 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id KAA05928 for smp-outgoing; Sun, 2 Nov 1997 10:45:56 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from mail.cs.tu-berlin.de (root@mail.cs.tu-berlin.de [130.149.17.13]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id KAA05919 for ; Sun, 2 Nov 1997 10:45:45 -0800 (PST) (envelope-from wosch@cs.tu-berlin.de) Received: from panke.panke.de (anonymous221.ppp.cs.tu-berlin.de [130.149.17.221]) by mail.cs.tu-berlin.de (8.8.6/8.8.7) with ESMTP id TAA16318; Sun, 2 Nov 1997 19:42:03 +0100 (MET) Received: (from wosch@localhost) by panke.panke.de (8.8.5/8.6.12) id SAA00510; Sun, 2 Nov 1997 18:21:50 +0100 (MET) To: "Jordan K. Hubbard" Cc: Wolfram Schneider , smp@freebsd.org Subject: Re: Some SMP timing tests. References: <5168.878410338@time.cdrom.com> From: Wolfram Schneider Date: 02 Nov 1997 18:21:48 +0100 In-Reply-To: "Jordan K. Hubbard"'s message of Sat, 01 Nov 1997 10:52:18 -0800 Message-ID: Lines: 171 Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk "Jordan K. Hubbard" writes: > > If you have a Makefile with 5 targets (e.g. 4 *.c files, 1 manpage), > > make can only create 5 jobs at once. Not surprising ;-) > > A good point, but this doesn't quite explain the behavior during > compiling things like libc, which has many targets which could be > parallelized. The bottleneck is the SUBDIR target which is not parallelized. Try the patch and run `make -j -I share/mk all' in /usr/src. The patch split the SUBDIR variable into 4 variables SUBDIR1, ... SUBDIR4 and runs 4 subdir targets simultaneously. Index: Makefile =================================================================== RCS file: /usr/cvs/src/Makefile,v retrieving revision 1.155 diff -u -r1.155 Makefile --- Makefile 1997/10/10 13:02:36 1.155 +++ Makefile 1997/11/02 11:32:56 @@ -39,6 +39,11 @@ # Put initial settings here. SUBDIR= +SUBDIR1= +SUBDIR2= +SUBDIR3= +SUBDIR4= + # We must do share/info early so that installation of info `dir' # entries works correctly. Do it first since it is less likely to @@ -50,52 +55,52 @@ # We must do include and lib early so that the perl *.ph generation # works correctly as it uses the header files installed by this. .if exists(include) -SUBDIR+= include +SUBDIR2+= include .endif .if exists(lib) -SUBDIR+= lib +SUBDIR2+= lib .endif .if exists(bin) -SUBDIR+= bin +SUBDIR3+= bin .endif .if exists(games) && !defined(NOGAMES) -SUBDIR+= games +SUBDIR1+= games .endif .if exists(gnu) -SUBDIR+= gnu +SUBDIR1+= gnu .endif .if exists(kerberosIV) && !defined(NOCRYPT) && defined(MAKE_KERBEROS4) -SUBDIR+= kerberosIV +SUBDIR2+= kerberosIV .endif .if exists(libexec) -SUBDIR+= libexec +SUBDIR2+= libexec .endif .if exists(sbin) -SUBDIR+= sbin +SUBDIR4+= sbin .endif .if exists(share) && !defined(NOSHARE) -SUBDIR+= share +SUBDIR3+= share .endif .if exists(sys) -SUBDIR+= sys +SUBDIR4+= sys .endif .if exists(usr.bin) -SUBDIR+= usr.bin +SUBDIR3+= usr.bin .endif .if exists(usr.sbin) -SUBDIR+= usr.sbin +SUBDIR4+= usr.sbin .endif .if exists(secure) && !defined(NOCRYPT) && !defined(NOSECURE) -SUBDIR+= secure +SUBDIR2+= secure .endif .if exists(lkm) && !defined(NOLKM) -SUBDIR+= lkm +SUBDIR4+= lkm .endif # etc must be last for "distribute" to work .if exists(etc) && make(distribute) -SUBDIR+= etc +SUBDIR4+= etc .endif # These are last, since it is nice to at least get the base system Index: share/mk/bsd.subdir.mk =================================================================== RCS file: /usr/cvs/src/share/mk/bsd.subdir.mk,v retrieving revision 1.18 diff -u -r1.18 bsd.subdir.mk --- bsd.subdir.mk 1997/06/21 15:40:34 1.18 +++ bsd.subdir.mk 1997/11/02 11:29:28 @@ -46,7 +46,57 @@ cd ${.CURDIR}/$${edir}; \ fi; \ ${MAKE} ${.TARGET:realinstall=install} DIRPRFX=${DIRPRFX}$$edir/); \ - done + done; \ + for entry in ${SUBDIR1}; do \ + (if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ + ${ECHODIR} "===> ${DIRPRFX}$${entry}.${MACHINE}"; \ + edir=$${entry}.${MACHINE}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}$$entry"; \ + edir=$${entry}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${.TARGET:realinstall=install} DIRPRFX=${DIRPRFX}$$edir/); \ + done & \ + for entry in ${SUBDIR2}; do \ + (if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ + ${ECHODIR} "===> ${DIRPRFX}$${entry}.${MACHINE}"; \ + edir=$${entry}.${MACHINE}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}$$entry"; \ + edir=$${entry}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${.TARGET:realinstall=install} DIRPRFX=${DIRPRFX}$$edir/); \ + done & \ + for entry in ${SUBDIR3}; do \ + (if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ + ${ECHODIR} "===> ${DIRPRFX}$${entry}.${MACHINE}"; \ + edir=$${entry}.${MACHINE}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}$$entry"; \ + edir=$${entry}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${.TARGET:realinstall=install} DIRPRFX=${DIRPRFX}$$edir/); \ + done & \ + for entry in ${SUBDIR}; do \ + (if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ + ${ECHODIR} "===> ${DIRPRFX}$${entry}.${MACHINE}"; \ + edir=$${entry}.${MACHINE}; \ + cd ${.CURDIR}/$${edir}; \ + else \ + ${ECHODIR} "===> ${DIRPRFX}$$entry"; \ + edir=$${entry}; \ + cd ${.CURDIR}/$${edir}; \ + fi; \ + ${MAKE} ${.TARGET:realinstall=install} DIRPRFX=${DIRPRFX}$$edir/); \ + done & \ + wait + ${SUBDIR}:: @if test -d ${.TARGET}.${MACHINE}; then \ -- Wolfram Schneider http://www.apfel.de/~wosch/ From owner-freebsd-smp Sun Nov 2 12:20:59 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id MAA10465 for smp-outgoing; Sun, 2 Nov 1997 12:20:59 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id MAA10455 for ; Sun, 2 Nov 1997 12:20:55 -0800 (PST) (envelope-from jkh@time.cdrom.com) Received: from time.cdrom.com (jkh@localhost.cdrom.com [127.0.0.1]) by time.cdrom.com (8.8.7/8.6.9) with ESMTP id MAA01236; Sun, 2 Nov 1997 12:20:53 -0800 (PST) To: Wolfram Schneider cc: smp@freebsd.org Subject: Re: Some SMP timing tests. In-reply-to: Your message of "02 Nov 1997 18:21:48 +0100." Date: Sun, 02 Nov 1997 12:20:52 -0800 Message-ID: <1232.878502052@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > The bottleneck is the SUBDIR target which is not parallelized. > > Try the patch and run `make -j -I share/mk all' in /usr/src. > The patch split the SUBDIR variable into 4 variables SUBDIR1, > ... SUBDIR4 and runs 4 subdir targets simultaneously. Oh *igitt*! There has to be a better way than this, I think. What about having bsd.subdir.mk take an arbitrary ${SUBDIR} and just divide it up internally? Having to manually break your subdir list up in to 4 categories really loses as a general mechanism, though I will indeed try your patch temporarily just to see what effects it has on build time. :) Thanks. Jordan From owner-freebsd-smp Sun Nov 2 12:57:10 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id MAA11751 for smp-outgoing; Sun, 2 Nov 1997 12:57:10 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from ns.mt.sri.com (SRI-56K-FR.mt.net [206.127.65.42]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id MAA11745 for ; Sun, 2 Nov 1997 12:57:05 -0800 (PST) (envelope-from nate@rocky.mt.sri.com) Received: from rocky.mt.sri.com (rocky.mt.sri.com [206.127.76.100]) by ns.mt.sri.com (8.8.7/8.8.7) with ESMTP id NAA10659; Sun, 2 Nov 1997 13:56:48 -0700 (MST) Received: (from nate@localhost) by rocky.mt.sri.com (8.7.5/8.7.3) id NAA03962; Sun, 2 Nov 1997 13:56:45 -0700 (MST) Date: Sun, 2 Nov 1997 13:56:45 -0700 (MST) Message-Id: <199711022056.NAA03962@rocky.mt.sri.com> From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: "Jordan K. Hubbard" Cc: Wolfram Schneider , smp@freebsd.org Subject: Re: Some SMP timing tests. In-Reply-To: <1232.878502052@time.cdrom.com> References: <1232.878502052@time.cdrom.com> X-Mailer: VM 6.29 under 19.15 XEmacs Lucid Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > The bottleneck is the SUBDIR target which is not parallelized. > > > > Try the patch and run `make -j -I share/mk all' in /usr/src. > > The patch split the SUBDIR variable into 4 variables SUBDIR1, > > ... SUBDIR4 and runs 4 subdir targets simultaneously. > > Oh *igitt*! There has to be a better way than this, I think. > > What about having bsd.subdir.mk take an arbitrary ${SUBDIR} and > just divide it up internally? Unfortunately, this won't work generically because of dependencies. Ie; we *must* build/install the include files before doing other things. Nate From owner-freebsd-smp Sun Nov 2 14:15:13 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id OAA15129 for smp-outgoing; Sun, 2 Nov 1997 14:15:13 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from picnic.mat.net (picnic.mat.net [206.246.122.117]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id OAA15124 for ; Sun, 2 Nov 1997 14:15:09 -0800 (PST) (envelope-from chuckr@glue.umd.edu) Received: from localhost (chuckr@localhost) by picnic.mat.net (8.8.7/8.8.5) with SMTP id QAA03920; Sun, 2 Nov 1997 16:12:48 -0500 (EST) X-Authentication-Warning: picnic.mat.net: chuckr owned process doing -bs Date: Sun, 2 Nov 1997 16:12:47 -0500 (EST) From: Chuck Robey X-Sender: chuckr@picnic.mat.net To: Wolfram Schneider cc: "Jordan K. Hubbard" , smp@freebsd.org Subject: Re: Some SMP timing tests. In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On 2 Nov 1997, Wolfram Schneider wrote: > "Jordan K. Hubbard" writes: > > > If you have a Makefile with 5 targets (e.g. 4 *.c files, 1 manpage), > > > make can only create 5 jobs at once. Not surprising ;-) > > > > A good point, but this doesn't quite explain the behavior during > > compiling things like libc, which has many targets which could be > > parallelized. > > The bottleneck is the SUBDIR target which is not parallelized. > > Try the patch and run `make -j -I share/mk all' in /usr/src. > The patch split the SUBDIR variable into 4 variables SUBDIR1, > ... SUBDIR4 and runs 4 subdir targets simultaneously. OK, I tried it. I started with no obj at all, did a make obj outside the timing loop, just so that I knew it was completely clean. This goes for both runs. The first one is before the patches were inserted, the second one after. Both on a Tyan Titan II, 2 each PPro 166's e/w 512K cache each, 64 Megs main memory. I have set the source on the first disk, the obj on the second, and swap evenly distributed between the two. Here's the results: before patching-> /usr/bin/time -l make -j 12 buildworld: 5389.14 real 3229.98 user 2902.01 sys 8200 maximum resident set size 249 average shared memory size 306 average unshared data size 58 average unshared stack size 7096181 page reclaims 2930 page faults 0 swaps 34729 block input operations 209264 block output operations 0 messages sent 0 messages received 8 signals received 903820 voluntary context switches 430529 involuntary context switches With patched Makefile and bsd.subdir.mk, same command: 2780.55 real 1696.39 user 1571.73 sys 6828 maximum resident set size 270 average shared memory size 350 average unshared data size 60 average unshared stack size 4079428 page reclaims 2607 page faults 0 swaps 10619 block input operations 130129 block output operations 0 messages sent 0 messages received 0 signals received 473969 voluntary context switches 227592 involuntary context switches Pretty major difference? Seems to me .... ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@glue.umd.edu | communications topic, C programming, and Unix. 213 Lakeside Drive Apt T-1 | Greenbelt, MD 20770 | I run Journey2 and picnic, both FreeBSD (301) 220-2114 | version 3.0 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Sun Nov 2 16:55:19 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id QAA22456 for smp-outgoing; Sun, 2 Nov 1997 16:55:19 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from mail.cs.tu-berlin.de (root@mail.cs.tu-berlin.de [130.149.17.13]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id QAA22434 for ; Sun, 2 Nov 1997 16:55:12 -0800 (PST) (envelope-from wosch@cs.tu-berlin.de) Received: from panke.panke.de (anonymous213.ppp.cs.tu-berlin.de [130.149.17.213]) by mail.cs.tu-berlin.de (8.8.6/8.8.7) with ESMTP id BAA25664; Mon, 3 Nov 1997 01:50:13 +0100 (MET) Received: (from wosch@localhost) by panke.panke.de (8.8.5/8.6.12) id BAA01606; Mon, 3 Nov 1997 01:48:05 +0100 (MET) To: "Jordan K. Hubbard" Cc: Wolfram Schneider , smp@freebsd.org Subject: Re: Some SMP timing tests. References: <1232.878502052@time.cdrom.com> From: Wolfram Schneider Date: 03 Nov 1997 01:48:01 +0100 In-Reply-To: "Jordan K. Hubbard"'s message of Sun, 02 Nov 1997 12:20:52 -0800 Message-ID: Lines: 21 Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk "Jordan K. Hubbard" writes: > > The bottleneck is the SUBDIR target which is not parallelized. > > Try the patch and run `make -j -I share/mk all' in /usr/src. > > The patch split the SUBDIR variable into 4 variables SUBDIR1, > > ... SUBDIR4 and runs 4 subdir targets simultaneously. > > Oh *igitt*! There has to be a better way than this, I think. Well, run du /usr/src | xdu and compare the size of the subdir 'etc' and 'gnu' ;-/ You must balance the tree by hand. > What about having bsd.subdir.mk take an arbitrary ${SUBDIR} and > just divide it up internally? Having to manually break your subdir > list up in to 4 categories really loses as a general mechanism, -- Wolfram Schneider http://www.apfel.de/~wosch/ From owner-freebsd-smp Mon Nov 3 00:22:31 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id AAA16699 for smp-outgoing; Mon, 3 Nov 1997 00:22:31 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from mail.cs.tu-berlin.de (mail.cs.tu-berlin.de [130.149.17.13]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id AAA16687 for ; Mon, 3 Nov 1997 00:22:13 -0800 (PST) (envelope-from wosch@cs.tu-berlin.de) Received: from panke.panke.de (anonymous214.ppp.cs.tu-berlin.de [130.149.17.214]) by mail.cs.tu-berlin.de (8.8.6/8.8.7) with ESMTP id JAA22267; Mon, 3 Nov 1997 09:14:16 +0100 (MET) Received: (from wosch@localhost) by panke.panke.de (8.8.5/8.6.12) id JAA00434; Mon, 3 Nov 1997 09:10:12 +0100 (MET) To: Chuck Robey Cc: Wolfram Schneider , "Jordan K. Hubbard" , smp@freebsd.org Subject: Re: Some SMP timing tests. References: From: Wolfram Schneider Date: 03 Nov 1997 09:10:10 +0100 In-Reply-To: Chuck Robey's message of Sun, 2 Nov 1997 16:12:47 -0500 (EST) Message-ID: Lines: 20 Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Chuck Robey writes: > OK, I tried it. I started with no obj at all, did a make obj outside the > timing loop, just so that I knew it was completely clean. This goes for > both runs. The first one is before the patches were inserted, the second > one after. Both on a Tyan Titan II, 2 each PPro 166's e/w 512K cache > each, 64 Megs main memory. I have set the source on the first disk, the > obj on the second, and swap evenly distributed between the two. Here's > the results: > > before patching-> > > /usr/bin/time -l make -j 12 buildworld: > 5389.14 real 3229.98 user 2902.01 sys > With patched Makefile and bsd.subdir.mk, same command: > 2780.55 real 1696.39 user 1571.73 sys I doubt that the patch can reduce the time up to 50%. -- Wolfram Schneider http://www.apfel.de/~wosch/ From owner-freebsd-smp Mon Nov 3 00:25:00 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id AAA16947 for smp-outgoing; Mon, 3 Nov 1997 00:25:00 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from time.cdrom.com (root@time.cdrom.com [204.216.27.226]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id AAA16930 for ; Mon, 3 Nov 1997 00:24:57 -0800 (PST) (envelope-from jkh@time.cdrom.com) Received: from time.cdrom.com (jkh@localhost.cdrom.com [127.0.0.1]) by time.cdrom.com (8.8.7/8.6.9) with ESMTP id AAA08147; Mon, 3 Nov 1997 00:24:37 -0800 (PST) To: Wolfram Schneider cc: Chuck Robey , smp@freebsd.org Subject: Re: Some SMP timing tests. In-reply-to: Your message of "03 Nov 1997 09:10:10 +0100." Date: Mon, 03 Nov 1997 00:24:37 -0800 Message-ID: <8144.878545477@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > /usr/bin/time -l make -j 12 buildworld: > > 5389.14 real 3229.98 user 2902.01 sys > > With patched Makefile and bsd.subdir.mk, same command: > > 2780.55 real 1696.39 user 1571.73 sys > > I doubt that the patch can reduce the time up to 50%. Hmmm. Are you sure this works with the world target? I applied the patch and ran a make world with it, and it died here: ===> lib ===> lib/csu/i386 install -C -o bin -g bin -m 444 /usr/src/lib/csu/i386/dlfcn.h /usr/include install -c -o bin -g bin -m 444 crt0.o c++rt0.o gcrt0.o scrt0.o sgcrt0.o /usr/l ib install: crt0.o: No such file or directory *** Error code 71 Stop. [During the "make all install" phase] Jordan From owner-freebsd-smp Mon Nov 3 07:52:00 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id HAA09767 for smp-outgoing; Mon, 3 Nov 1997 07:52:00 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from picnic.mat.net (picnic.mat.net [206.246.122.117]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id HAA09756 for ; Mon, 3 Nov 1997 07:51:53 -0800 (PST) (envelope-from chuckr@glue.umd.edu) Received: from localhost (chuckr@localhost) by picnic.mat.net (8.8.7/8.8.5) with SMTP id JAA08871; Mon, 3 Nov 1997 09:45:36 -0500 (EST) X-Authentication-Warning: picnic.mat.net: chuckr owned process doing -bs Date: Mon, 3 Nov 1997 09:45:35 -0500 (EST) From: Chuck Robey X-Sender: chuckr@picnic.mat.net To: "Jordan K. Hubbard" cc: Wolfram Schneider , smp@freebsd.org Subject: Re: Some SMP timing tests. In-Reply-To: <8144.878545477@time.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Mon, 3 Nov 1997, Jordan K. Hubbard wrote: > > > /usr/bin/time -l make -j 12 buildworld: > > > 5389.14 real 3229.98 user 2902.01 sys > > > With patched Makefile and bsd.subdir.mk, same command: > > > 2780.55 real 1696.39 user 1571.73 sys > > > > I doubt that the patch can reduce the time up to 50%. > > Hmmm. Are you sure this works with the world target? I applied > the patch and ran a make world with it, and it died here: > > ===> lib > ===> lib/csu/i386 > install -C -o bin -g bin -m 444 /usr/src/lib/csu/i386/dlfcn.h /usr/include > install -c -o bin -g bin -m 444 crt0.o c++rt0.o gcrt0.o scrt0.o sgcrt0.o /usr/l > ib > install: crt0.o: No such file or directory > *** Error code 71 > > Stop. > > [During the "make all install" phase] I only did the make buildworld, not an install, I dunno. > > Jordan > > ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@glue.umd.edu | communications topic, C programming, and Unix. 213 Lakeside Drive Apt T-1 | Greenbelt, MD 20770 | I run Journey2 and picnic, both FreeBSD (301) 220-2114 | version 3.0 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Mon Nov 3 07:53:52 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id HAA09868 for smp-outgoing; Mon, 3 Nov 1997 07:53:52 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from picnic.mat.net (picnic.mat.net [206.246.122.117]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id HAA09854 for ; Mon, 3 Nov 1997 07:53:46 -0800 (PST) (envelope-from chuckr@glue.umd.edu) Received: from localhost (chuckr@localhost) by picnic.mat.net (8.8.7/8.8.5) with SMTP id JAA08860; Mon, 3 Nov 1997 09:44:24 -0500 (EST) X-Authentication-Warning: picnic.mat.net: chuckr owned process doing -bs Date: Mon, 3 Nov 1997 09:44:23 -0500 (EST) From: Chuck Robey X-Sender: chuckr@picnic.mat.net To: Wolfram Schneider cc: "Jordan K. Hubbard" , smp@freebsd.org Subject: Re: Some SMP timing tests. In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On 3 Nov 1997, Wolfram Schneider wrote: > Chuck Robey writes: > > OK, I tried it. I started with no obj at all, did a make obj outside the > > timing loop, just so that I knew it was completely clean. This goes for > > both runs. The first one is before the patches were inserted, the second > > one after. Both on a Tyan Titan II, 2 each PPro 166's e/w 512K cache > > each, 64 Megs main memory. I have set the source on the first disk, the > > obj on the second, and swap evenly distributed between the two. Here's > > the results: > > > > before patching-> > > > > /usr/bin/time -l make -j 12 buildworld: > > 5389.14 real 3229.98 user 2902.01 sys > > With patched Makefile and bsd.subdir.mk, same command: > > 2780.55 real 1696.39 user 1571.73 sys > > I doubt that the patch can reduce the time up to 50%. I wondered about that too, but I made the conditions as identical as I could. I wonder what I missed? > > -- > Wolfram Schneider http://www.apfel.de/~wosch/ > > ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@glue.umd.edu | communications topic, C programming, and Unix. 213 Lakeside Drive Apt T-1 | Greenbelt, MD 20770 | I run Journey2 and picnic, both FreeBSD (301) 220-2114 | version 3.0 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Mon Nov 3 11:05:08 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id LAA24430 for smp-outgoing; Mon, 3 Nov 1997 11:05:08 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.5.84]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id LAA24419 for ; Mon, 3 Nov 1997 11:05:01 -0800 (PST) (envelope-from tlambert@usr09.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.8.7/8.8.7) id MAA10686; Mon, 3 Nov 1997 12:04:26 -0700 (MST) Received: from usr09.primenet.com(206.165.6.209) via SMTP by smtp03.primenet.com, id smtpd010637; Mon Nov 3 12:04:17 1997 Received: (from tlambert@localhost) by usr09.primenet.com (8.8.5/8.8.5) id MAA20435; Mon, 3 Nov 1997 12:04:14 -0700 (MST) From: Terry Lambert Message-Id: <199711031904.MAA20435@usr09.primenet.com> Subject: Re: Some SMP timing tests. To: jkh@time.cdrom.com (Jordan K. Hubbard) Date: Mon, 3 Nov 1997 19:04:13 +0000 (GMT) Cc: wosch@cs.tu-berlin.de, smp@freebsd.org In-Reply-To: <1232.878502052@time.cdrom.com> from "Jordan K. Hubbard" at Nov 2, 97 12:20:52 pm X-Mailer: ELM [version 2.4 PL23] Content-Type: text Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > The bottleneck is the SUBDIR target which is not parallelized. > > > > Try the patch and run `make -j -I share/mk all' in /usr/src. > > The patch split the SUBDIR variable into 4 variables SUBDIR1, > > ... SUBDIR4 and runs 4 subdir targets simultaneously. > > Oh *igitt*! There has to be a better way than this, I think. > > What about having bsd.subdir.mk take an arbitrary ${SUBDIR} and > just divide it up internally? Having to manually break your subdir > list up in to 4 categories really loses as a general mechanism, > though I will indeed try your patch temporarily just to see what > effects it has on build time. :) I think this isn't done because make's topological sort for finding dependencies only works across a single instance of the workspace. This means that if it spawned an inferior make, you would have to fix all implicit dependencies that are currently expressed as the ordering of the SUBDIR lines. Specifically, you would have to make a dependency for "libc.so" that you enforced it getting made before anything which linked with it, and so on, for each and every library. The "includes" stuff is also expressed as an implied order (really, the Makefiles in the source tree are in bad shape as regards having all of the dependencies explicitly listed). You could handle this the good way, or you could do it the cheesy way. The good way would be to be able to specify directory dependencies in the makefiles including the subdir.mk so that the order was known to the make descending, and it could parallelize as much as possible so long as it preserved dependency order. Of course, libc itself would not have any ordering requirements, and that would fix the parallel builds there. The cheesy way would be to make a "PSUBDIRS", and let the directories there be descended in parallel only after the SUBDIRS targets had been made, in order. This would also fix the libc builds the way you want. This approach is cheesy because it doesn't admit that there might be a directory A build that depends on a directory B build, and directories C,D and E,F with the same dependency relationship. The good soloution would descend A,C,E simultaneously. The cheesy one could only do A,C,E in order, and then descend B,D,F simultaneously. This is the simplest failure mode for the cheesy soloution -- for greater depth with dependency, the failure to parallelize increases exponentially. Again with the libc, you could conceive of wanting to build the things which are statically linked after libc.a is built but before libc.so has finished. Likewise, you could decide to seperate build and link targets, and build all the sources, but only link the static once once libc.a has been build, and the dynamic ones when libc.so has been built, etc.. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Nov 3 11:09:37 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id LAA24860 for smp-outgoing; Mon, 3 Nov 1997 11:09:37 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from mail.cs.tu-berlin.de (root@mail.cs.tu-berlin.de [130.149.17.13]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id LAA24853 for ; Mon, 3 Nov 1997 11:09:34 -0800 (PST) (envelope-from wosch@cs.tu-berlin.de) Received: from panke.panke.de (anonymous230.ppp.cs.tu-berlin.de [130.149.17.230]) by mail.cs.tu-berlin.de (8.8.6/8.8.7) with ESMTP id UAA11067; Mon, 3 Nov 1997 20:03:24 +0100 (MET) Received: (from wosch@localhost) by panke.panke.de (8.8.5/8.6.12) id TAA00456; Mon, 3 Nov 1997 19:07:22 +0100 (MET) To: Chuck Robey Cc: Wolfram Schneider , "Jordan K. Hubbard" , smp@freebsd.org Subject: Re: Some SMP timing tests. References: From: Wolfram Schneider Date: 03 Nov 1997 19:07:18 +0100 In-Reply-To: Chuck Robey's message of Mon, 3 Nov 1997 09:44:23 -0500 (EST) Message-ID: Lines: 14 Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Chuck Robey writes: > > > /usr/bin/time -l make -j 12 buildworld: > > > 5389.14 real 3229.98 user 2902.01 sys > > > With patched Makefile and bsd.subdir.mk, same command: > > > 2780.55 real 1696.39 user 1571.73 sys > > I doubt that the patch can reduce the time up to 50%. > I wondered about that too, but I made the conditions as identical as I > could. I wonder what I missed? Did you compare the size of the obj trees? Both should have the same size... -- Wolfram Schneider http://www.apfel.de/~wosch/ From owner-freebsd-smp Mon Nov 3 15:13:44 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id PAA12117 for smp-outgoing; Mon, 3 Nov 1997 15:13:44 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from fedex.austx.tandem.com (fedex.austx.tandem.com [131.124.250.27]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id PAA12101 for ; Mon, 3 Nov 1997 15:13:29 -0800 (PST) (envelope-from larsf@austx.tandem.com) Received: from jake.austx.tandem.com (jake-gw.austx.tandem.com [131.124.4.53]) by fedex.austx.tandem.com (8.8.6/8.8.0) with ESMTP id RAA25546; Mon, 3 Nov 1997 17:12:47 -0600 (CST) Received: (from larsf@localhost) by jake.austx.tandem.com (8.8.5/8.7.1) id RAA18714; Mon, 3 Nov 1997 17:12:46 -0600 (CST) Message-Id: <199711032312.RAA18714@jake.austx.tandem.com> Subject: p6dne and latest bios To: smp@freebsd.org Date: Mon, 3 Nov 1997 17:12:45 -0600 (CST) From: "Lars Fredriksen" Cc: larsf@austx.tandem.com Reply-To: fredriks@mcs.com X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, I just loaded the latest bios on my p6dne supermicro board, and now it doesn't boot. It gets past the point of installing the scsi bioses, but then hangs tight. ctrl-alt-del doesn't work, it doesn't access the floppy or anything. It did boot once, but that was it, and that is out of 20-30 tries. Any combination of settings in the bios doesn't seem to make any difference. I have loaded the fail-safe options, disabled booting from anything but the floppy or other, disabled the internal ide and so far no luck to make it boot(the os) reliably. Anyone else see this or something similar? Sincerely, Lars -- ------------------------------------------------------------------- Lars Fredriksen larsf@mpd.tandem.com From owner-freebsd-smp Mon Nov 3 17:11:32 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id RAA20076 for smp-outgoing; Mon, 3 Nov 1997 17:11:32 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from sendero-ppp.i-connect.net (sendero-ppp.i-Connect.Net [206.190.143.100]) by hub.freebsd.org (8.8.7/8.8.7) with SMTP id RAA20036 for ; Mon, 3 Nov 1997 17:11:20 -0800 (PST) (envelope-from shimon@sendero-ppp.i-connect.net) Received: (qmail 22549 invoked by uid 1000); 4 Nov 1997 01:10:56 -0000 Message-ID: X-Mailer: XFMail 1.2-beta-103097 [p0] on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 In-Reply-To: <199711032312.RAA18714@jake.austx.tandem.com> Date: Mon, 03 Nov 1997 17:10:56 -0800 (PST) Organization: Atlas Telecom From: Simon Shapiro To: fredriks@mcs.com Subject: RE: p6dne and latest bios Cc: larsf@austx.tandem.com, smp@freebsd.org Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi Lars Fredriksen; On 03-Nov-97 you wrote: > Hi, > I just loaded the latest bios on my p6dne supermicro board, > and now it doesn't boot. It gets past the point of installing the scsi > bioses, but then hangs tight. ctrl-alt-del doesn't work, it doesn't > access the floppy or anything. It did boot once, but that was it, and > that is out of 20-30 tries. Any combination of settings in the bios > doesn't seem to make any difference. I have loaded the fail-safe > options, > disabled booting from anything but the floppy or other, disabled the > internal ide and so far no luck to make it boot(the os) reliably. > > Anyone else see this or something similar? Yup. SuperMicro is fast turning into trash. We have invested THOUSANDS in their motherboards to find they do not run SMP correctly. Support is so sad it is funny. --- If Microsoft Built Cars: Every time they repainted the lines on the road, you'd have to buy a new car. Sincerely Yours, Simon Shapiro Atlas Telecom Senior Architect 14355 SW Allen Blvd., Suite 130 Beaverton OR 97005 Shimon@i-Connect.Net Voice: 503.799.2313 From owner-freebsd-smp Tue Nov 4 22:53:33 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id WAA17972 for smp-outgoing; Tue, 4 Nov 1997 22:53:33 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from fredriks-1.pr.mcs.net (fredriks-1.pr.mcs.net [205.164.50.241]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id WAA17967 for ; Tue, 4 Nov 1997 22:53:30 -0800 (PST) (envelope-from lars@fredriks-1.pr.mcs.net) Received: (from lars@localhost) by fredriks-1.pr.mcs.net (8.8.7/8.6.6) id AAA00485 for smp@freebsd.org; Wed, 5 Nov 1997 00:53:20 -0600 (CST) From: Lars Fredriksen Message-Id: <199711050653.AAA00485@fredriks-1.pr.mcs.net> Subject: p6dne problems fixed To: smp@freebsd.org Date: Wed, 5 Nov 1997 00:53:20 -0600 (CST) X-Mailer: ELM [version 2.4ME+ PL28 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, Just a quick update to anyone. The bios file that is associated with the P6DNE on supermicro's web site (na7081.zip) does not handle 2 Ppros. If you accidentially burn your flash with this rom, you need to remove the second PPro, and then reburn with na0904.rom (techsupport sent me this file but was not able to tell me the above :-()) or later. Needless to say I am back up :-) By the way steve, this is what my mptable now looks like: =============================================================================== MPTable, version 2.0.15 ------------------------------------------------------------------------------- MP Floating Pointer Structure: location: BIOS physical address: 0x000fb940 signature: '_MP_' length: 16 bytes version: 1.4 checksum: 0x64 mode: Virtual Wire ------------------------------------------------------------------------------- MP Config Table Header: physical address: 0x000f7db0 signature: 'PCMP' base table length: 284 version: 1.4 checksum: 0xd6 OEM ID: 'INTEL ' Product ID: '440FX ' OEM table pointer: 0x00000000 OEM table size: 0 entry count: 27 local APIC address: 0xfee00000 extended table length: 8 extended table checksum: 117 ------------------------------------------------------------------------------- MP Config Base Table Entries: -- Processors: APIC ID Version State Family Model Step Flags 0 0x11 BSP, usable 6 1 9 0xfbff 1 0x11 AP, usable 6 1 7 0xfbff -- Bus: Bus ID Type 0 PCI 1 ISA -- I/O APICs: APIC ID Version State Address 2 0x11 usable 0xfec00000 -- I/O Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# ExtINT conforms conforms 1 0 2 0 INT conforms conforms 1 1 2 1 INT conforms conforms 1 0 2 2 INT conforms conforms 1 3 2 3 INT conforms conforms 1 4 2 4 INT conforms conforms 1 6 2 6 INT conforms conforms 1 7 2 7 INT active-hi edge 1 8 2 8 INT conforms conforms 1 9 2 9 INT conforms conforms 1 10 2 10 INT conforms conforms 1 12 2 12 INT conforms conforms 1 13 2 13 INT conforms conforms 1 15 2 20 INT active-lo level 0 18:A 2 16 INT active-lo level 0 17:A 2 16 INT active-lo level 0 18:A 2 17 INT active-lo level 0 17:A 2 17 INT active-lo level 0 19:A 2 18 INT active-lo level 0 20:A 2 19 SMI conforms conforms 1 0 2 23 -- Local Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# ExtINT conforms conforms 0 0:A 255 0 NMI conforms conforms 0 0:A 255 1 ------------------------------------------------------------------------------- MP Config Extended Table Entries: -- bus ID: 1 bus info: 0x01 parent bus ID: 0 ------------------------------------------------------------------------------- # SMP kernel config file options: # Required: options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O # Optional (built-in defaults will work in most cases): #options NCPU=2 # number of CPUs #options NBUS=2 # number of busses #options NAPIC=1 # number of IO APICs #options NINTR=24 # number of INTs =============================================================================== -- ------------------------------------------------------------------- Lars Fredriksen fredriks@mcs.com (home) lars@fredriks-1.pr.mcs.net (home-home) From owner-freebsd-smp Wed Nov 5 10:40:10 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id KAA27909 for smp-outgoing; Wed, 5 Nov 1997 10:40:10 -0800 (PST) (envelope-from owner-freebsd-smp) Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [205.168.119.129]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id KAA27894 for ; Wed, 5 Nov 1997 10:40:04 -0800 (PST) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost [127.0.0.1]) by Ilsa.StevesCafe.com (8.8.7/8.8.5) with ESMTP id LAA28666; Wed, 5 Nov 1997 11:07:06 -0700 (MST) Message-Id: <199711051807.LAA28666@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0gamma 1/27/96 From: Steve Passe To: Lars Fredriksen cc: smp@freebsd.org Subject: Re: p6dne problems fixed In-reply-to: Your message of "Wed, 05 Nov 1997 00:53:20 CST." <199711050653.AAA00485@fredriks-1.pr.mcs.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 05 Nov 1997 11:07:06 -0700 Sender: owner-freebsd-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Hi, > Just a quick update to anyone. The bios file that is associated with > the P6DNE on supermicro's web site (na7081.zip) does not handle 2 Ppros. If > you accidentially burn your flash with this rom, you need to remove the second > PPro, and then reburn with na0904.rom (techsupport sent me this file but was > not able to tell me the above :-()) or later. > > Needless to say I am back up :-) > > By the way steve, this is what my mptable now looks like: > ... > -- > I/O Ints: Type Polarity Trigger Bus ID IRQ APIC ID PIN# > ExtINT conforms conforms 1 0 2 0 > INT conforms conforms 1 1 2 1 > INT conforms conforms 1 0 2 2 > INT conforms conforms 1 3 2 3 > INT conforms conforms 1 4 2 4 > INT conforms conforms 1 6 2 6 > INT conforms conforms 1 7 2 7 > INT active-hi edge 1 8 2 8 > INT conforms conforms 1 9 2 9 > INT conforms conforms 1 10 2 10 > INT conforms conforms 1 12 2 12 > INT conforms conforms 1 13 2 13 > INT conforms conforms 1 15 2 20 > INT active-lo level 0 18:A 2 16 > INT active-lo level 0 17:A 2 16 > INT active-lo level 0 18:A 2 17 > INT active-lo level 0 17:A 2 17 > INT active-lo level 0 19:A 2 18 > INT active-lo level 0 20:A 2 19 > SMI conforms conforms 1 0 2 23 I dont have time for an analysis right now, but does anyone else see a problem in the above table??? -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD