From owner-freebsd-smp Sun Nov 7 13: 8:52 1999 Delivered-To: freebsd-smp@freebsd.org Received: from verdi.nethelp.no (verdi.nethelp.no [158.36.41.162]) by hub.freebsd.org (Postfix) with SMTP id 8753914D08 for ; Sun, 7 Nov 1999 13:08:47 -0800 (PST) (envelope-from sthaug@nethelp.no) Received: (qmail 17701 invoked by uid 1001); 7 Nov 1999 21:08:44 +0000 (GMT) To: freebsd-smp@FreeBSD.ORG, Soren Schmidt Subject: Re: Dual Celeron + FreeBSD? From: sthaug@nethelp.no In-Reply-To: Your message of "Sat, 06 Nov 1999 21:39:13 +0100" References: <98447.941920753@verdi.nethelp.no> X-Mailer: Mew version 1.05+ on Emacs 19.34.2 Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Date: Sun, 07 Nov 1999 22:08:44 +0100 Message-ID: <17699.942008924@verdi.nethelp.no> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I wrote: > Below are some bonnie *and* rawio measurements, with comments. After > the first few tests of bonnie with 100 MB files I stopped doing this > test, since it was obvious that the memory size of the machine had a > great effect - even on the seek transactions. I stuck to 1000 MB test > afterwards - this is greater than the memory on all the machines I > tried. Did a few more tests after I found the correct ATA-66 cable. The disk is now connected to the HPT366 controller on the Abit BP6 board (instead of the PIIX4 controller). There are actually a few differences to be seen. The previous bonnie figures for UDMA33 are included for comparison. > 3. Dual Celeron-366, 66 MHz system bus, 256 MB memory, > IBM DPTA-372730 (EIDE), 4.0-CURRENT as of 29. October, new ATA driver. > > Typical wall clock time 453 s for 1000 MB (30 s longer than 3.3 with old > wd driver). But the block output figures are better :-) > > 1000 16430 96.1 17374 45.0 5547 16.1 15941 95.3 22465 35.6 123.8 3.1 > 1000 16040 94.1 17447 44.7 5557 15.7 15796 95.2 22452 35.5 122.9 3.0 > 1000 16374 95.4 17508 45.6 5566 15.9 15804 95.0 22434 35.6 121.8 3.0 > 1000 15724 95.4 17753 43.5 5558 15.7 15849 95.6 22408 35.2 121.3 3.1 > 1000 16415 96.0 17428 44.3 5561 15.7 15855 95.5 22451 35.5 122.1 3.0 Same but UDMA66. Typical wall clock time 448 s (5 s lower than UDMA33): 1000 15966 97.2 19324 50.2 5593 16.7 15857 95.0 22474 35.3 121.4 3.1 1000 16269 95.5 19275 48.8 5592 16.3 15888 95.2 22461 35.1 119.5 3.1 1000 15832 95.4 19296 50.8 5561 16.3 15884 95.4 22447 35.5 121.8 3.0 1000 16113 94.7 19266 51.3 5591 16.5 15960 95.5 22440 35.7 120.0 2.9 1000 15728 94.8 19248 49.9 5556 16.3 15913 95.4 22413 35.4 121.7 3.0 1000 15388 96.0 19464 48.7 5572 16.0 15715 95.4 22420 35.2 121.6 3.0 > 4. Same except single Celeron-366. > > 1000 17496 93.0 17420 34.0 5603 11.7 17290 94.0 22465 22.7 120.1 2.0 > 1000 17330 91.7 17466 33.4 5572 11.8 17302 94.1 22467 22.9 121.5 2.0 > 1000 17388 91.3 17554 32.0 5564 11.4 17145 93.4 22375 22.4 122.5 2.1 > 1000 17239 92.1 17486 31.8 5565 11.3 17300 94.4 22442 22.5 121.6 2.1 > 1000 17492 90.7 17627 31.5 5575 10.9 17251 93.9 22462 22.5 121.4 2.0 > 1000 17756 91.8 17544 33.5 5557 11.7 17219 93.7 22443 23.1 122.3 1.9 Single Celeron-33 and UDMA66. Typical wall clock time 438 s. 1000 17845 94.5 19864 39.1 5593 12.7 17085 92.7 22466 22.6 125.0 2.2 1000 18219 94.8 19522 38.1 5579 12.3 17085 92.5 22460 22.7 122.3 2.1 1000 17685 93.9 19649 38.1 5568 12.3 17103 93.1 22404 22.8 121.5 1.9 1000 17675 94.1 19296 40.9 5577 13.4 17289 94.0 22446 23.2 122.7 2.0 1000 17735 93.8 19236 37.1 5591 12.4 17273 94.1 22458 22.9 122.0 1.9 Some rawio figures for the single Celeron-366, UDMA66 case. It is only sequential read that show any marked difference (more than 53000 kB/s). Notice also that two of the runs produced sequential write bandwidth that was only about half of the other runs. I also had one run which made the machine hang solid - couldn't break to DDB or anything. This indicates to me that there may still be bugs in the HPT366 support. Single Celeron-366, UDMA66 RR anon 1734.9 107 0.1 0.3 0.4 16384 SR anon 53726.0 3279 0.7 6.8 7.5 16384 RW anon 1575.9 97 0.1 0.2 0.3 16384 SW anon 14373.4 877 0.2 1.9 2.1 16384 RR anon 1719.8 107 0.1 0.3 0.4 16384 SR anon 53753.1 3281 0.8 7.2 8.0 16384 RW anon 1564.3 97 0.1 0.2 0.3 16384 SW anon 14334.5 875 0.3 1.8 2.1 16384 RR anon 1736.4 108 0.1 0.3 0.4 16384 SR anon 53769.0 3282 0.9 7.1 8.0 16384 RW anon 1565.2 98 0.1 0.3 0.3 16384 SW anon 14211.9 867 0.1 2.0 2.1 16384 RR anon 1723.1 108 0.1 0.3 0.4 16384 SR anon 53793.9 3283 1.1 6.8 7.9 16384 RW anon 1571.3 97 0.1 0.3 0.3 16384 SW anon 14270.4 871 0.1 2.1 2.1 16384 Two strange cases with lower sequential write bandwidth: Test ID K/sec /sec %User %Sys %Total RR anon 1743.8 107 0.1 0.3 0.4 16384 SR anon 53735.4 3280 0.4 7.7 8.1 16384 RW anon 1567.7 97 0.1 0.2 0.3 16384 SW anon 7532.2 460 0.2 0.9 1.2 16384 RR anon 1740.9 108 0.1 0.3 0.4 16384 SR anon 53745.0 3280 0.8 7.0 7.8 16384 RW anon 1574.7 97 0.1 0.3 0.3 16384 SW anon 7553.1 461 0.1 1.0 1.1 16384 Steinar Haug, Nethelp consulting, sthaug@nethelp.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Nov 8 9:51: 8 1999 Delivered-To: freebsd-smp@freebsd.org Received: from dingo.cdrom.com (castles517.castles.com [208.214.165.81]) by hub.freebsd.org (Postfix) with ESMTP id 4D6C115239 for ; Mon, 8 Nov 1999 09:51:02 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Received: from dingo.cdrom.com (localhost [127.0.0.1]) by dingo.cdrom.com (8.9.3/8.8.8) with ESMTP id JAA19666; Mon, 8 Nov 1999 09:41:56 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Message-Id: <199911081741.JAA19666@dingo.cdrom.com> X-Mailer: exmh version 2.0.2 2/24/98 To: james.g.mansion@hsbcgroup.com Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Dual Celeron + FreeBSD? In-reply-to: Your message of "Mon, 08 Nov 1999 11:04:41 GMT." <80256823.003D4AD1.00@hbmdtesmtp1.markets.uk.hsbc> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 08 Nov 1999 09:41:56 -0800 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Aside from the various issues about serialised compiling > in some parts of the tree, UDMA vs SCSI bun fight, etc, > I'm interested that you are claiming memory bandwidth is > the bottleneck. Well, when you're no longer talking to any peripherals, memory bandwidth is the only limiting factor. A 'make world' entirely from/to MFS can only be limited by memory bandwidth. > I've seen claims that Intel chips are memory bandwidth > limited for as long as I can remember but the expected > brick walls have been elusive, whatever the very low > level benchmarks say, and the scalability of e.g. SQLServer > and other databases across quads and higher seems quite > good, which is somewhat counter to your suggestion which > as I read it implies that a quad Xeon is already starting to > hit a brick wall for non-trivial (in-cache) code and 8-way > systems are on a hiding to nothing. Er, so what you're saying is that while the low-level benchmarks demonstrate there's a limit, the fact that the high-level benchmarks keep getting faster means that the low-level benchmarks must be wrong? Has it ever occurred to you that the high-level benchmarks might still be limited by other factors? > Is it possible to use the CPUs' counters to measure how > long they spend busy-waiting on the giant lock and record > it somewhere once they enter the kernel itself? Probably. It's certainly demonstrate that we would benefit a lot from more kernel reentrancy in thos particular benchmark. > Is it not the case that even with the lazy write feature turned > on the single threaded kernel (without even a possibility of > a task switch when blocked on entry) will tend to make a > system that is doing a lot of small IOs against a memory-based > file system or cache look CPU or memory bound (from the > lock spins) rather than IO bound? My understanding was > that gcc+binutils compiles are far from optimal in terms of > their IO pattern wrt minimising the number of kernel entries. I'm sure they're not, but we're not talking about optimal performance here, we're talking about the behaviour of a specific code set. > I'd be concerned about pointing the finger at the CPU/memory > interface based on measurements from a system which has > such a busy-wait mechanism, unless there is other supporting > evidence (ideally compute-bound application benchmarks, > rather than low-level memory stress tests). Perhaps I wasn't being clear enough; in the situations I'm considering, the only limiting factor available is memory bandwidth. No, our architecture isn't taking advantage of the parallelism available, but the serialisation points are also limited basically by memory bandwidth. > wrt 'make -j', some of the levels of paralellisation being used > (e.g. Adam Strohl using -j 12) would surely be entirely > counter-productive with soft updates or a memory file system, > since you are more likely to swap, more likely to have a whole > bunch of processes spinning uselessly on the lock, making the > scheduler work harder, and don't actually have much physical > IO time to fill in with computing anyway? You're unlikely to swap in most of these cases, actually. The memory footprint for -j12 builds is surprisingly small, and in my case I'm using a system with 1GB of memory. The scheduler doesn't "work harder", since fortunately our scheduler is fairly close to O(1) in behaviour. But your point about the GKL is quite valid. -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Nov 9 14:41:55 1999 Delivered-To: freebsd-smp@freebsd.org Received: from orion.ac.hmc.edu (Orion.AC.HMC.Edu [134.173.32.20]) by hub.freebsd.org (Postfix) with ESMTP id 9530214CF0 for ; Tue, 9 Nov 1999 14:41:41 -0800 (PST) (envelope-from brdavis@orion.ac.hmc.edu) Received: (from brdavis@localhost) by orion.ac.hmc.edu (8.8.8/8.8.8) id OAA10745 for freebsd-smp@freebsd.org; Tue, 9 Nov 1999 14:41:37 -0800 (PST) Date: Tue, 9 Nov 1999 14:41:37 -0800 From: Brooks Davis To: freebsd-smp@freebsd.org Subject: AMI MegaPlex II Message-ID: <19991109144137.A5948@orion.ac.hmc.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0pre4i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, I'm looking at building a cluster of quad proc Xeons using FreeBSD. I'm wondering if anyone has any experience good or bad with the AMI MegaPlex II system. Since these nodes won't have much disk the 4U version is very attractive looking. They also have 64-bit PCI which would be nice for use with Myrinet. We're funded to so we need to choose some hardware and start getting quotes. Thanks, Brooks To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 10 10:58:23 1999 Delivered-To: freebsd-smp@freebsd.org Received: from bofh.dermak.pl (bofh.dermak.pl [212.160.174.2]) by hub.freebsd.org (Postfix) with ESMTP id 6D3FB14E16 for ; Wed, 10 Nov 1999 10:57:52 -0800 (PST) (envelope-from jacke@dermak.pl) Received: from localhost (jacke@localhost) by bofh.dermak.pl (8.9.3/8.9.3) with ESMTP id TAA62936; Wed, 10 Nov 1999 19:56:46 +0100 (CET) (envelope-from jacke@dermak.pl) Date: Wed, 10 Nov 1999 19:56:44 +0100 (CET) From: Jakub Klausa To: Mike Smith Cc: Tamiji Homma , freebsd-smp@FreeBSD.ORG Subject: Re: Dual Celeron + FreeBSD? In-Reply-To: <199911060210.SAA01384@dingo.cdrom.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org -----BEGIN PGP SIGNED MESSAGE----- On Fri, 5 Nov 1999, Mike Smith wrote: => From experience to date, I find that fairly hard to believe. Most => people that have claimed otherwise so far have subsequently been found => or, or ignored. 8) Ok. So you've said it took 35 minutes orso for a quad processor Xenon machine to make world with everything in MFS (except the destination place for new binaries). Now, I was really surprised by such a poor results of your tests. I've tried to build today's RELENG_3 on my machine. Some lusers were working, but not too hard (some e-mail reading, news reading, irc and such - not too much or resources used by them). Look at the result: >>> elf make world started on Wed Nov 10 17:14:33 CET 1999 >>> elf make world completed on Wed Nov 10 17:57:19 CET 1999 Which's less then 43 minutes. Quite a time considering that's only dual processor machine. It's a stock source sompilation, without any additional parameteras, tweaks or somesuch. Plain 'make world -j32' was what I issued. BTW, the hardware in question is: CPU: 2x PIII450 (_NOT_ Xenons) MB : Tyan Thunderbolt HDD: da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled (sources and destination on the same physical disk, softupdates enabled) The disk was connected to the onboard controller: ahc0: rev 0x00 int a irq 16 on pci0.11.0 RAM: 512MB (I intetnionally didn't make any MFS tweaks or such). Hm... makes me wonder how could some of the people on the list say that the Tyan MBs are not good at all. I once again made myself up that was one of the best chices i've maid. - -- k. -----BEGIN PGP SIGNATURE----- Version: 2.6.3ia Charset: noconv iQCVAwUBOCm/7k54q5P/Kvv1AQGHYQP/YGGwZwNejVBIUFnuySIY1WOmTlCJqZMQ 3tkrFK/8lTIWZn2Oemx9BDkW9ALVrF3Np9zNOzgZcww2ivmghcDZ8COJomYob60C lkPH7mGpZ7y8smIcCXa13/0wJxSTO98w3HiqiEcIg+lQHEs9AphryRa5uS8r9Rta jm70awNmZy0= =Tf1j -----END PGP SIGNATURE----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Nov 10 11:17:50 1999 Delivered-To: freebsd-smp@freebsd.org Received: from bud.itixs.com (bud.itixs.com [216.186.144.67]) by hub.freebsd.org (Postfix) with SMTP id 58137152A4 for ; Wed, 10 Nov 1999 11:17:34 -0800 (PST) (envelope-from bifrost@itixs.com) Received: (qmail 527 invoked from network); 10 Nov 1999 00:31:45 -0000 Received: from fortress.itixs.com (@10.1.1.5) by bud.itixs.com with SMTP; 10 Nov 1999 00:31:45 -0000 Date: Tue, 9 Nov 1999 16:31:36 -0800 (PST) From: Tom Sparks To: Brooks Davis Cc: freebsd-smp@FreeBSD.ORG Subject: Re: AMI MegaPlex II In-Reply-To: <19991109144137.A5948@orion.ac.hmc.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 9 Nov 1999, Brooks Davis wrote: > I'm looking at building a cluster of quad proc Xeons using FreeBSD. I'm > wondering if anyone has any experience good or bad with the AMI MegaPlex > II system. Since these nodes won't have much disk the 4U version is > very attractive looking. They also have 64-bit PCI which would be nice > for use with Myrinet. We're funded to so we need to choose some > hardware and start getting quotes. I used one of AMI's quad PPro boxes with great success, unfortunately it was axed before we got FreeBSD-SMP running (I think a VP's kid stole it). I would say go for it, the only other manufacturers with 4 way boxes that I've seen are Compaq, Intel, and I'm sure GW2k and Dell have some decent boxes. 4U is *very* small for a 4way box so definately go for it :) Let me know how it works, I may be looking for similar hardware where I'm at. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Nov 11 8:59:33 1999 Delivered-To: freebsd-smp@freebsd.org Received: from dingo.cdrom.com (castles524.castles.com [208.214.165.88]) by hub.freebsd.org (Postfix) with ESMTP id E712F14C05 for ; Thu, 11 Nov 1999 08:59:27 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Received: from dingo.cdrom.com (localhost [127.0.0.1]) by dingo.cdrom.com (8.9.3/8.8.8) with ESMTP id IAA00711; Thu, 11 Nov 1999 08:47:57 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Message-Id: <199911111647.IAA00711@dingo.cdrom.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Jakub Klausa Cc: Mike Smith , Tamiji Homma , freebsd-smp@FreeBSD.ORG Subject: Re: Dual Celeron + FreeBSD? In-reply-to: Your message of "Wed, 10 Nov 1999 19:56:44 +0100." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 11 Nov 1999 08:47:56 -0800 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > -----BEGIN PGP SIGNED MESSAGE----- > > On Fri, 5 Nov 1999, Mike Smith wrote: > > => From experience to date, I find that fairly hard to believe. Most > => people that have claimed otherwise so far have subsequently been found > => or, or ignored. 8) > > Ok. So you've said it took 35 minutes orso for a quad processor Xenon > machine to make world with everything in MFS (except the destination place > for new binaries). > > Now, I was really surprised by such a poor results of your tests. I've > tried to build today's RELENG_3 on my machine. Some lusers were working, > but not too hard (some e-mail reading, news reading, irc and such - not > too much or resources used by them). You cannot compare apples and oranges. This box is building -current. -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ msmith@freebsd.org \\ and he'll hate you for a lifetime. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message