From owner-freebsd-smp@FreeBSD.ORG Thu May 5 05:04:55 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F4A016A4CE for ; Thu, 5 May 2005 05:04:55 +0000 (GMT) Received: from ylpvm01.prodigy.net (ylpvm01-ext.prodigy.net [207.115.57.32]) by mx1.FreeBSD.org (Postfix) with ESMTP id 749FD43D6D for ; Thu, 5 May 2005 05:04:54 +0000 (GMT) (envelope-from ketralnis@ketralnis.dyndns.org) Received: from ketralnis.dyndns.org (adsl-64-166-85-255.dsl.sntc01.pacbell.net [64.166.85.255])j4554HXw029488 for ; Thu, 5 May 2005 01:04:18 -0400 Received: from marshall ([10.0.0.249]) by ketralnis.dyndns.org (8.13.1/8.13.1) with SMTP id j4554csO041603 for ; Wed, 4 May 2005 22:04:38 -0700 (PDT) (envelope-from ketralnis@ketralnis.dyndns.org) Message-ID: <001b01c5512f$c190eb20$f900000a@marshall> From: "David King" To: Date: Wed, 4 May 2005 22:03:19 -0700 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 Subject: HP Netserver LT 6000r X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: David King List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 May 2005 05:04:55 -0000 Has anyone had any luck getting an HP Netserver LT 6000r to run on FreeBSD, with SMP working? I am looking at buying a quad 700MHZ of this model and want to make sure it will run my primary operating system. The only posts I could readily find were failures several years ago, so I hope the situation has improved :) From owner-freebsd-smp@FreeBSD.ORG Fri May 6 15:44:59 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D14D716A4D5; Fri, 6 May 2005 15:44:59 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9EAAF43D41; Fri, 6 May 2005 15:44:59 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id E110C512B1; Fri, 6 May 2005 08:44:58 -0700 (PDT) Date: Fri, 6 May 2005 08:44:58 -0700 From: Kris Kennaway To: sparc64@FreeBSD.org, smp@FreeBSD.org Message-ID: <20050506154458.GA13055@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wac7ysb48OaltWcw" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: 'panic: spin lock held too long' at boot X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 15:45:00 -0000 --wac7ysb48OaltWcw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline The following has happened twice recently while booting 6.0 on a 12-processor e4500: [...] Waiting 8 seconds for SCSI devices to settle SMP: AP CPU #11 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! panic: spin lock held too long0xffffff813e774730 foor >5 seecons cpuid = 11 KDB: enter: panic [hangs] Any ideas? Kris --wac7ysb48OaltWcw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe5D6Wry0BWjoQKURAiOhAKDp/hyydbTxiY1y25VQrkLnQO72RQCeMC0F i+8Dh1crRlL2g7eVvqGB4WY= =jZ59 -----END PGP SIGNATURE----- --wac7ysb48OaltWcw-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 18:35:31 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F143216A4D4; Fri, 6 May 2005 18:35:30 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7068143D53; Fri, 6 May 2005 18:35:30 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 7FBD6513B6; Fri, 6 May 2005 11:35:29 -0700 (PDT) Date: Fri, 6 May 2005 11:35:29 -0700 From: Kris Kennaway To: smp@FreeBSD.org, current@FreeBSD.org Message-ID: <20050506183529.GA46411@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LZvS9be/3tNcYl/X" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Subject: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 18:35:31 -0000 --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Here are my benchmark numbers for parallel tarball extraction with/without mpsafevfs on a 12-processor E4500 running up-to-date 6.0. Kernel was built without INVARIANTS and other debugging options, without ADAPTIVE_GIANT (which causes about a 200% performance penalty on system time in my testing, and has marginal impact on real or user time) and with 4BSD scheduler (ULE causes spontaneous reboots on this machine). The e4500 uses the esp SCSI controller, which runs without Giant. The test is this: #!/bin/sh for i in 1 2 3 4 5 6 7 8 9 10 11 12; do mkdir $i tar xfC /var/portbuild/sparc64/5/tarballs/bindist.tar $i & done on a 2000mb preallocated malloc backed md disk (machine has 5GB RAM). Before each test I umount, newfs with default options (i.e. no -U; this kills performance on md by a factor of several times) and mount. The tarball is # ls -l /var/portbuild/sparc64/5/tarballs/bindist.tar -rw-r--r-- 1 kris kris 133231104 Apr 28 12:18 /var/portbuild/sparc64/5/tarballs/bindist.tar # tar tvf /var/portbuild/sparc64/5/tarballs/bindist.tar | wc -l 5664 (it's a copy of a sparc64 5.4-STABLE world I use to populate package build chroots). A single extraction (with tarball cached) with mpsafevfs=1 takes: 14.85 real 1.31 user 10.43 sys 14.90 real 1.31 user 10.40 sys 15.03 real 1.26 user 10.55 sys 14.49 real 1.35 user 10.47 sys 14.50 real 1.36 user 10.42 sys 14.50 real 1.28 user 10.52 sys 14.52 real 1.33 user 10.48 sys 14.44 real 1.38 user 10.36 sys 14.54 real 1.37 user 10.39 sys 14.63 real 1.29 user 10.56 sys mean=14.64 seconds real time without mpsafevfs: 14.72 real 1.39 user 10.45 sys 14.70 real 1.40 user 10.47 sys 14.99 real 1.41 user 10.54 sys 15.13 real 1.48 user 10.45 sys 15.18 real 1.40 user 10.50 sys 14.87 real 1.64 user 10.38 sys 14.66 real 1.42 user 10.37 sys 14.69 real 1.49 user 10.30 sys 14.87 real 1.45 user 10.60 sys 14.75 real 1.47 user 10.43 sys mean=14.86 real x mpsafevfs + !mpsafevfs +--------------------------------------------------------------------------+ | + x | | + ++ + + + x xx x x + x + x + x x| ||__________M________A__|_________________|_M________________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 10 14.66 15.18 14.87 14.856 0.18810163 + 10 14.44 15.03 14.54 14.64 0.2081666 Difference at 95.0% confidence -0.216 +/- 0.186404 -1.45396% +/- 1.25474% (Student's t, pooled s = 0.198388) So mpsafevfs has a slight measurable benefit even for non-concurrent extraction. The parallel extraction without mpsafevfs: 319.42 real 35.70 user 1547.38 sys 317.80 real 35.41 user 1532.87 sys 318.49 real 35.35 user 1542.23 sys 321.82 real 35.51 user 1559.50 sys 317.66 real 35.51 user 1566.16 sys 318.63 real 35.64 user 1552.48 sys 319.51 real 35.69 user 1548.99 sys 317.79 real 35.34 user 1542.89 sys 319.89 real 35.70 user 1536.34 sys 318.76 real 35.24 user 1545.21 sys with mpsafevfs: 80.24 real 27.70 user 475.54 sys 83.13 real 27.94 user 491.55 sys 87.66 real 28.45 user 500.68 sys 81.88 real 28.12 user 463.51 sys 83.23 real 27.87 user 483.62 sys 82.20 real 28.07 user 482.57 sys 83.82 real 28.29 user 473.70 sys 84.54 real 27.95 user 472.12 sys 80.29 real 28.24 user 461.87 sys 87.77 real 28.34 user 482.03 sys 82.10 real 27.79 user 475.31 sys system clock: +--------------------------------------------------------------------------+ | x ++ | | x ++ | |xx +++| |xxxx +++| ||A| |A | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 10 461.87 500.68 482.03 478.719 11.975802 + 10 1532.87 1566.16 1547.38 1547.405 10.066401 Difference at 95.0% confidence 1068.69 +/- 10.3942 223.239% +/- 2.17124% (Student's t, pooled s = 11.0624) wall clock: +--------------------------------------------------------------------------+ | + | | + | | + | | x + | | x + | | x + | |xx + | |xxx + | |xxx ++| ||A| A|| +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 11 80.24 87.77 83.13 83.350909 2.5277241 + 10 317.66 321.82 318.76 318.977 1.2618333 Difference at 95.0% confidence 235.626 +/- 1.85556 282.692% +/- 2.2262% (Student's t, pooled s = 2.02905) i.e. mpsafevfs shows enormous improvements in both cases. Comparing to the mean time for a single extraction, 12 simultaneous extractions with mpsafevfs take the time of 5.69 single, and 21.788 without mpsafevfs. This is an effective concurrency of 2.11 (12/5.69) extractions for mpsafevfs and 0.55 without (i.e. nearly twice as bad as just sequentializing the extractions). I might be bumping into the bandwidth of md here - when I ran less rigorous tests with lower concurrency of extractions I seemed to be getting marginally better performance (about an effective concurrency of 2.2 for both 3 and 10 simultaneous extractions - so at least it doesn't seem to degrade badly). Or this might be reflecting VFS lock contention (which there is certainly a lot of, according to mutex profiling traces). Certainly for package builds on this machine I get much better performance and lower CPU utilization if I do every package build in a separate (swap-backed) md than with them all in a single large md, which tells me it's not hard to saturate a single md. Even if I am hitting another limit here that is placing an upper bound on the performance, filesystem performance with mpsafevfs is clearly much better than without, and we are now seeing clear benefits from SMP on 6.0 compared to earlier versions of FreeBSD. Kris P.S. Big props to Jeff Roberson for making this work! Thanks also to Hiroki Sato for donating the E4500 and other machine resources. --LZvS9be/3tNcYl/X Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe7jwWry0BWjoQKURAlH/AKCMw4PwgfxoeVa2CG06L5EYHe2QxACeOM+z 7s556CrOB/lNGi1w4KxQ+gg= =T+/F -----END PGP SIGNATURE----- --LZvS9be/3tNcYl/X-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 18:43:55 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE14016A4D4; Fri, 6 May 2005 18:43:55 +0000 (GMT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 686B043D6A; Fri, 6 May 2005 18:43:55 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.13.0/8.13.0) with ESMTP id j46IhsxQ012893; Fri, 6 May 2005 11:43:54 -0700 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.13.0/8.13.0/Submit) id j46Ihs8P012892; Fri, 6 May 2005 11:43:54 -0700 Date: Fri, 6 May 2005 11:43:54 -0700 From: Brooks Davis To: Kris Kennaway Message-ID: <20050506184354.GA12366@odin.ac.hmc.edu> References: <20050506183529.GA46411@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rwEMma7ioTxnRzrJ" Content-Disposition: inline In-Reply-To: <20050506183529.GA46411@xor.obsecurity.org> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new X-Spam-Status: No, hits=0.0 required=8.0 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on odin.ac.hmc.edu cc: smp@freebsd.org cc: current@freebsd.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 18:43:55 -0000 --rwEMma7ioTxnRzrJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > Here are my benchmark numbers for parallel tarball extraction > with/without mpsafevfs on a 12-processor E4500 running up-to-date 6.0. > Kernel was built without INVARIANTS and other debugging options, > without ADAPTIVE_GIANT (which causes about a 200% performance penalty > on system time in my testing, and has marginal impact on real or user > time) and with 4BSD scheduler (ULE causes spontaneous reboots on this > machine). The e4500 uses the esp SCSI controller, which runs > without Giant. Excellect writeup! Thanks for doing this. Many thanks to Jeff! Some of my users will certaintly thank you when we upgrade our cluster to 6.x. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --rwEMma7ioTxnRzrJ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFCe7rqXY6L6fI4GtQRAjz0AKCtPu0OHvjI2ks84Evlwex9DbQEIgCfSuGm hIDMcQwa4THTlEJM8ug+9HI= =3uEM -----END PGP SIGNATURE----- --rwEMma7ioTxnRzrJ-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 18:48:53 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8C93116A4D4; Fri, 6 May 2005 18:48:53 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 56F7143D68; Fri, 6 May 2005 18:48:53 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 89AB252048; Fri, 6 May 2005 11:48:52 -0700 (PDT) Date: Fri, 6 May 2005 11:48:52 -0700 From: Kris Kennaway To: Kris Kennaway Message-ID: <20050506184852.GA62656@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jRHKVT23PllUwdXP" Content-Disposition: inline In-Reply-To: <20050506183529.GA46411@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i cc: smp@FreeBSD.org cc: current@FreeBSD.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 18:48:53 -0000 --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > I might be bumping into the bandwidth of md here - when I ran less > rigorous tests with lower concurrency of extractions I seemed to be > getting marginally better performance (about an effective concurrency > of 2.2 for both 3 and 10 simultaneous extractions - so at least it > doesn't seem to degrade badly). Or this might be reflecting VFS lock > contention (which there is certainly a lot of, according to mutex > profiling traces). I suspect that I am hitting the md bandwidth: # dd if=/dev/zero of=/dev/md0 bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) which is a lot worse than I expected (even for a 400MHz CPU). For some reason I get better performance writing to a filesystem mounted on this md: # dd if=/dev/zero of=foo bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 7.943042 secs (66005946 bytes/sec) # rm foo # dd if=/dev/zero of=foo bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 7.126929 secs (73564364 bytes/sec) # rm foo # dd if=/dev/zero of=foo bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes transferred in 7.237668 secs (72438804 bytes/sec) If the write bandwidth is only 50-70MB/sec, then it won't be hard to saturate, so I won't probe the full scalability of mpsafevfs here. Kris --jRHKVT23PllUwdXP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe7wUWry0BWjoQKURAjXSAJsEEw3Y4ZbPatJUiKBe6tvp8tiE0wCfaqSa DW+OVE3ZLxL/NCKrxB5ToIk= =vojN -----END PGP SIGNATURE----- --jRHKVT23PllUwdXP-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 19:05:56 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0D93C16A4D4 for ; Fri, 6 May 2005 19:05:56 +0000 (GMT) Received: from mail.rfnj.org (ns1.rfnj.org [66.180.172.156]) by mx1.FreeBSD.org (Postfix) with ESMTP id A08AD43D1D for ; Fri, 6 May 2005 19:05:55 +0000 (GMT) (envelope-from asym@rfnj.org) Received: from megalomaniac.rfnj.org (ool-45736df1.dyn.optonline.net [69.115.109.241]) by mail.rfnj.org (Postfix) with ESMTP id ED7562B6; Fri, 6 May 2005 15:05:55 -0400 (EDT) Message-Id: <6.2.1.2.2.20050506150138.036b83c0@mail.rfnj.org> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Fri, 06 May 2005 15:11:39 -0400 To: Kris Kennaway From: Allen In-Reply-To: <20050506184852.GA62656@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506184852.GA62656@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: smp@FreeBSD.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 19:05:56 -0000 At 14:48 5/6/2005, Kris Kennaway wrote: >On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > > > I might be bumping into the bandwidth of md here - when I ran less > > rigorous tests with lower concurrency of extractions I seemed to be > > getting marginally better performance (about an effective concurrency > > of 2.2 for both 3 and 10 simultaneous extractions - so at least it > > doesn't seem to degrade badly). Or this might be reflecting VFS lock > > contention (which there is certainly a lot of, according to mutex > > profiling traces). > >I suspect that I am hitting the md bandwidth: > ># dd if=/dev/zero of=/dev/md0 bs=1024k count=500 >500+0 records in >500+0 records out >524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) > >which is a lot worse than I expected (even for a 400MHz CPU). That looks pretty goofy. Even PC66 SDRAM should be able to push ~250MB/s on a very slow processor. Does the md code for raw access really load this much work onto the CPU?? >For some reason I get better performance writing to a filesystem >mounted on this md: Part of me wants to think that maybe this is due to some fashion of metadata caching, in the manner of softupdates. Possible? From owner-freebsd-smp@FreeBSD.ORG Fri May 6 19:12:24 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ACDC416A4D4 for ; Fri, 6 May 2005 19:12:24 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7116043DA2 for ; Fri, 6 May 2005 19:12:24 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 9DD65514C9; Fri, 6 May 2005 12:12:23 -0700 (PDT) Date: Fri, 6 May 2005 12:12:23 -0700 From: Kris Kennaway To: Allen Message-ID: <20050506191223.GB70068@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506184852.GA62656@xor.obsecurity.org> <6.2.1.2.2.20050506150138.036b83c0@mail.rfnj.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1LKvkjL3sHcu1TtY" Content-Disposition: inline In-Reply-To: <6.2.1.2.2.20050506150138.036b83c0@mail.rfnj.org> User-Agent: Mutt/1.4.2.1i cc: smp@FreeBSD.org cc: Kris Kennaway Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 19:12:24 -0000 --1LKvkjL3sHcu1TtY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 06, 2005 at 03:11:39PM -0400, Allen wrote: > At 14:48 5/6/2005, Kris Kennaway wrote: > >On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > > > >> I might be bumping into the bandwidth of md here - when I ran less > >> rigorous tests with lower concurrency of extractions I seemed to be > >> getting marginally better performance (about an effective concurrency > >> of 2.2 for both 3 and 10 simultaneous extractions - so at least it > >> doesn't seem to degrade badly). Or this might be reflecting VFS lock > >> contention (which there is certainly a lot of, according to mutex > >> profiling traces). > > > >I suspect that I am hitting the md bandwidth: > > > ># dd if=3D/dev/zero of=3D/dev/md0 bs=3D1024k count=3D500 > >500+0 records in > >500+0 records out > >524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) > > > >which is a lot worse than I expected (even for a 400MHz CPU). >=20 > That looks pretty goofy. Even PC66 SDRAM should be able to push ~250MB/s= =20 > on a very slow processor. Does the md code for raw access really load th= is=20 > much work onto the CPU?? It must be something specific to the machine or architecture, because others have posted md speeds an order of magnitude faster than this (e.g. on amd64). > >For some reason I get better performance writing to a filesystem > >mounted on this md: >=20 > Part of me wants to think that maybe this is due to some fashion of=20 > metadata caching, in the manner of softupdates. Possible? Well, softupdates is disabled, but maybe there's something else going on. Kris --1LKvkjL3sHcu1TtY Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe8GXWry0BWjoQKURApIvAKDLP3b883E2/rBRQK0VPAc4AvAndwCggYs4 llGOYKcE8TYGA0J/4UiMDo8= =Z5zz -----END PGP SIGNATURE----- --1LKvkjL3sHcu1TtY-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 20:06:37 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE24416A4D6; Fri, 6 May 2005 20:06:37 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6892F43D5E; Fri, 6 May 2005 20:06:37 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 243E6514C9; Fri, 6 May 2005 13:06:36 -0700 (PDT) Date: Fri, 6 May 2005 13:06:35 -0700 From: Kris Kennaway To: Kris Kennaway Message-ID: <20050506200635.GB79102@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506184852.GA62656@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0ntfKIWw70PvrIHh" Content-Disposition: inline In-Reply-To: <20050506184852.GA62656@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i cc: smp@FreeBSD.org cc: current@FreeBSD.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 20:06:37 -0000 --0ntfKIWw70PvrIHh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 06, 2005 at 11:48:52AM -0700, Kris Kennaway wrote: > On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: >=20 > > I might be bumping into the bandwidth of md here - when I ran less > > rigorous tests with lower concurrency of extractions I seemed to be > > getting marginally better performance (about an effective concurrency > > of 2.2 for both 3 and 10 simultaneous extractions - so at least it > > doesn't seem to degrade badly). Or this might be reflecting VFS lock > > contention (which there is certainly a lot of, according to mutex > > profiling traces). >=20 > I suspect that I am hitting the md bandwidth: >=20 > # dd if=3D/dev/zero of=3D/dev/md0 bs=3D1024k count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) >=20 > which is a lot worse than I expected (even for a 400MHz CPU). >=20 > For some reason I get better performance writing to a filesystem > mounted on this md: >=20 > # dd if=3D/dev/zero of=3Dfoo bs=3D1024k count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 7.943042 secs (66005946 bytes/sec) > # rm foo > # dd if=3D/dev/zero of=3Dfoo bs=3D1024k count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 7.126929 secs (73564364 bytes/sec) > # rm foo > # dd if=3D/dev/zero of=3Dfoo bs=3D1024k count=3D500 > 500+0 records in > 500+0 records out > 524288000 bytes transferred in 7.237668 secs (72438804 bytes/sec) >=20 > If the write bandwidth is only 50-70MB/sec, then it won't be hard to > saturate, so I won't probe the full scalability of mpsafevfs here. I tried on a quad amd64 machine, which has md bandwidth an order of magnitude greater, but it has the same limiting concurrency of 2.2, so something else is happening here. Kris --0ntfKIWw70PvrIHh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe85LWry0BWjoQKURAuxLAJ9MLa6JtgUOCGLTL72JTpGU0zN1JACZAUiI H7bHiPx4WcQ90qsZ3aUVxKo= =SjNl -----END PGP SIGNATURE----- --0ntfKIWw70PvrIHh-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 20:12:05 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E5C4C16A4D4; Fri, 6 May 2005 20:12:05 +0000 (GMT) Received: from energistic.com (mail.energistic.com [216.54.148.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8C2C243D5A; Fri, 6 May 2005 20:12:05 +0000 (GMT) (envelope-from steve@energistic.com) Received: from energistic.com (steve@localhost.energistic.com [127.0.0.1]) by energistic.com (8.13.3/8.13.3) with ESMTP id j46KC45E014854; Fri, 6 May 2005 15:12:04 -0500 (EST) (envelope-from steve@energistic.com) Received: (from steve@localhost) by energistic.com (8.13.3/8.13.3/Submit) id j46KC4e9011926; Fri, 6 May 2005 15:12:04 -0500 (EST) (envelope-from steve) Date: Fri, 6 May 2005 15:12:04 -0500 From: Steve Ames To: Kris Kennaway Message-ID: <20050506201204.GA12764@energistic.com> References: <20050506183529.GA46411@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050506183529.GA46411@xor.obsecurity.org> User-Agent: Mutt/1.5.9i X-Spam-Status: No, score=-6.9 required=5.0 tests=AWL,BAYES_50,SPF_HELO_PASS, SPF_PASS,TW_EV,USER_IN_WHITELIST_TO autolearn=ham version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on energistic.com cc: smp@freebsd.org cc: current@freebsd.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 20:12:06 -0000 On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > Kernel was built without INVARIANTS and other debugging options, > without ADAPTIVE_GIANT (which causes about a 200% performance penalty > on system time in my testing, and has marginal impact on real or user > time) and with 4BSD scheduler (ULE causes spontaneous reboots on this > machine). The e4500 uses the esp SCSI controller, which runs > without Giant. I'm rather suprised that ADAPTIVE_GIANT causes so much performance degradation in your testing. On my, admittedly much more modest system, using ADAPTIVE_GIANT shaves about 6 minutes off a 'make world'. Haven't really done any research on why that is. I was just suprised by your 200% performance penalty. -Steve From owner-freebsd-smp@FreeBSD.ORG Fri May 6 20:26:27 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F244416A4D4; Fri, 6 May 2005 20:26:26 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9DB8843D9D; Fri, 6 May 2005 20:26:26 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id C5AE0513B6; Fri, 6 May 2005 13:26:25 -0700 (PDT) Date: Fri, 6 May 2005 13:26:25 -0700 From: Kris Kennaway To: Steve Ames Message-ID: <20050506202625.GA95077@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506201204.GA12764@energistic.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="n8g4imXOkfNTN/H1" Content-Disposition: inline In-Reply-To: <20050506201204.GA12764@energistic.com> User-Agent: Mutt/1.4.2.1i cc: smp@freebsd.org cc: current@freebsd.org cc: Kris Kennaway Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 20:26:27 -0000 --n8g4imXOkfNTN/H1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 06, 2005 at 03:12:04PM -0500, Steve Ames wrote: > On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > > Kernel was built without INVARIANTS and other debugging options, > > without ADAPTIVE_GIANT (which causes about a 200% performance penalty > > on system time in my testing, and has marginal impact on real or user > > time) and with 4BSD scheduler (ULE causes spontaneous reboots on this > > machine). The e4500 uses the esp SCSI controller, which runs > > without Giant. >=20 > I'm rather suprised that ADAPTIVE_GIANT causes so much performance > degradation in your testing. On my, admittedly much more modest system, > using ADAPTIVE_GIANT shaves about 6 minutes off a 'make world'. Haven't > really done any research on why that is. I was just suprised by your > 200% performance penalty. Note: system time, not wall clock time. It's probably because the system is otherwise largely Giant-free (no hardware devices under Giant, only things like ttys and CAM). On systems with more Giant usage in the I/O path you might see a benefit. Kris --n8g4imXOkfNTN/H1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe9LxWry0BWjoQKURAjz9AJwLtwR7YKSTx2em1+qbUsZmgqBm2ACfRYlv xgw1Kghp5JSq8lfP9LDDTDA= =2I9Q -----END PGP SIGNATURE----- --n8g4imXOkfNTN/H1-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 22:32:26 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 88BCB16A4D4; Fri, 6 May 2005 22:32:26 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id E6F6B43D6E; Fri, 6 May 2005 22:32:25 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id C840E514C9; Fri, 6 May 2005 15:32:24 -0700 (PDT) Date: Fri, 6 May 2005 15:32:24 -0700 From: Kris Kennaway To: Kris Kennaway Message-ID: <20050506223224.GA22379@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="r5Pyd7+fXNt84Ff3" Content-Disposition: inline In-Reply-To: <20050506183529.GA46411@xor.obsecurity.org> User-Agent: Mutt/1.4.2.1i cc: smp@FreeBSD.org cc: current@FreeBSD.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction (ULE vs 4BSD) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 22:32:26 -0000 --r5Pyd7+fXNt84Ff3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > Here are my benchmark numbers for parallel tarball extraction > with/without mpsafevfs on a 12-processor E4500 running up-to-date 6.0. > Kernel was built without INVARIANTS and other debugging options, > without ADAPTIVE_GIANT (which causes about a 200% performance penalty > on system time in my testing, and has marginal impact on real or user > time) and with 4BSD scheduler (ULE causes spontaneous reboots on this > machine). The e4500 uses the esp SCSI controller, which runs > without Giant. I tried with ULE on a quad amd64 machine, which was stable enough to perform the extraction tests. Here is the data for one tarball extraction to md: x real.one.4bsd + real.one.ule +--------------------------------------------------------------------------+ | + | | + | | + | |+ + + + x | |+ + + + x x x x x x x x| | |MA_| |___MA___| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 9 2.43 2.5 2.46 2.4644444 0.022973415 + 11 2.15 2.18 2.16 2.1636364 0.010269106 Difference at 95.0% confidence -0.300808 +/- 0.0161686 -12.2059% +/- 0.656073% (Student's t, pooled s = 0.0171217) ...so ULE is 12% faster at extracting a single tarball to md 12 concurrent extractions: x real.4bsd + real.ule +--------------------------------------------------------------------------+ | x x + + | |x x x x xx x x + + + ++ + + + +| | |____AM___| |______MA_______| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 10 13.56 14.43 14.09 14.02 0.2641969 + 11 15.95 17.34 16.49 16.570909 0.40957184 Difference at 95.0% confidence 2.55091 +/- 0.318571 18.1948% +/- 2.27226% (Student's t, pooled s = 0.348356) ...but 18% slower with 12 concurrent extractions. The effective concurrency under 4BSD is 2.11 (same asymptote as on the 12-processor sparc64, suggesting something universal like VFS locking is the limitation) but under ULE it is only 1.57. With 4 concurrent extractions (= # CPUs) x real.4.4bsd + real.4.ule +--------------------------------------------------------------------------+ | x + | |x xxx x x x * x + + + ++ + + +| | |____________AM____________| |_______A_M_____| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 10 4.65 5.44 5.17 5.157 0.22400645 + 10 5.41 5.87 5.69 5.657 0.1374409 Difference at 95.0% confidence 0.5 +/- 0.174609 9.69556% +/- 3.38587% (Student's t, pooled s = 0.185834) ULE is still slower. This suggests that at the present time (and apart from the known instabilities) ULE may be better for filesystem performance on lightly loaded systems, but it degrades worse than 4BSD under concurrent filesystem load. Kris --r5Pyd7+fXNt84Ff3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe/B3Wry0BWjoQKURAt8DAJ9V+Gjl7bJQRb4j6LqINP719SuZywCgx4ml EKziuNUiB2TLSU3PMmzaid8= =TZ8C -----END PGP SIGNATURE----- --r5Pyd7+fXNt84Ff3-- From owner-freebsd-smp@FreeBSD.ORG Fri May 6 23:00:15 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A9E7C16A4D6; Fri, 6 May 2005 23:00:15 +0000 (GMT) Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by mx1.FreeBSD.org (Postfix) with ESMTP id B270D43D68; Fri, 6 May 2005 23:00:14 +0000 (GMT) (envelope-from ticso@cicely12.cicely.de) Received: from cicely5.cicely.de (cicely5.cicely.de [10.1.1.7]) (authenticated bits=0)j46N084J075064 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=OK); Sat, 7 May 2005 01:00:11 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (cicely12.cicely.de [IPv6:3ffe:400:8d0:301::12]) by cicely5.cicely.de (8.12.10/8.12.10) with ESMTP id j46MxRhs023063 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 7 May 2005 00:59:28 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: from cicely12.cicely.de (localhost [127.0.0.1]) by cicely12.cicely.de (8.12.11/8.12.11) with ESMTP id j46MxRPR028934; Sat, 7 May 2005 00:59:27 +0200 (CEST) (envelope-from ticso@cicely12.cicely.de) Received: (from ticso@localhost) by cicely12.cicely.de (8.12.11/8.12.11/Submit) id j46MxROK028933; Sat, 7 May 2005 00:59:27 +0200 (CEST) (envelope-from ticso) Date: Sat, 7 May 2005 00:59:27 +0200 From: Bernd Walter To: Kris Kennaway Message-ID: <20050506225926.GB75629@cicely12.cicely.de> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506184852.GA62656@xor.obsecurity.org> <20050506200635.GB79102@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050506200635.GB79102@xor.obsecurity.org> X-Operating-System: FreeBSD cicely12.cicely.de 5.2-CURRENT alpha User-Agent: Mutt/1.5.6i X-Spam-Status: No, hits=-4.9 required=3.0 tests=BAYES_00 autolearn=no version=2.64 X-Spam-Report: * -4.9 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0042] X-Spam-Checker-Version: SpamAssassin 2.64 (2004-01-11) on cicely12.cicely.de cc: smp@freebsd.org cc: current@freebsd.org Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: ticso@cicely.de List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 23:00:15 -0000 On Fri, May 06, 2005 at 01:06:35PM -0700, Kris Kennaway wrote: > On Fri, May 06, 2005 at 11:48:52AM -0700, Kris Kennaway wrote: > > On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > > > > > I might be bumping into the bandwidth of md here - when I ran less > > > rigorous tests with lower concurrency of extractions I seemed to be > > > getting marginally better performance (about an effective concurrency > > > of 2.2 for both 3 and 10 simultaneous extractions - so at least it > > > doesn't seem to degrade badly). Or this might be reflecting VFS lock > > > contention (which there is certainly a lot of, according to mutex > > > profiling traces). > > > > I suspect that I am hitting the md bandwidth: > > > > # dd if=/dev/zero of=/dev/md0 bs=1024k count=500 > > 500+0 records in > > 500+0 records out > > 524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) Wasn't md's blocksize = systems's pagesize, IIRC 8k on sparc. I'm surprised that 1k even works. -- B.Walter BWCT http://www.bwct.de bernd@bwct.de info@bwct.de From owner-freebsd-smp@FreeBSD.ORG Fri May 6 23:09:54 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 45B3B16A4D4; Fri, 6 May 2005 23:09:54 +0000 (GMT) Received: from obsecurity.dyndns.org (CPE0050040655c8-CM00111ae02aac.cpe.net.cable.rogers.com [69.194.102.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id E92CD43DA1; Fri, 6 May 2005 23:09:53 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id A1C105221A; Fri, 6 May 2005 16:09:52 -0700 (PDT) Date: Fri, 6 May 2005 16:09:52 -0700 From: Kris Kennaway To: ticso@cicely.de Message-ID: <20050506230952.GB35398@xor.obsecurity.org> References: <20050506183529.GA46411@xor.obsecurity.org> <20050506184852.GA62656@xor.obsecurity.org> <20050506200635.GB79102@xor.obsecurity.org> <20050506225926.GB75629@cicely12.cicely.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="i0/AhcQY5QxfSsSZ" Content-Disposition: inline In-Reply-To: <20050506225926.GB75629@cicely12.cicely.de> User-Agent: Mutt/1.4.2.1i cc: smp@freebsd.org cc: current@freebsd.org cc: Kris Kennaway Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 23:09:54 -0000 --i0/AhcQY5QxfSsSZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 07, 2005 at 12:59:27AM +0200, Bernd Walter wrote: > On Fri, May 06, 2005 at 01:06:35PM -0700, Kris Kennaway wrote: > > On Fri, May 06, 2005 at 11:48:52AM -0700, Kris Kennaway wrote: > > > On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: > > >=20 > > > > I might be bumping into the bandwidth of md here - when I ran less > > > > rigorous tests with lower concurrency of extractions I seemed to be > > > > getting marginally better performance (about an effective concurren= cy > > > > of 2.2 for both 3 and 10 simultaneous extractions - so at least it > > > > doesn't seem to degrade badly). Or this might be reflecting VFS lo= ck > > > > contention (which there is certainly a lot of, according to mutex > > > > profiling traces). > > >=20 > > > I suspect that I am hitting the md bandwidth: > > >=20 > > > # dd if=3D/dev/zero of=3D/dev/md0 bs=3D1024k count=3D500 > > > 500+0 records in > > > 500+0 records out > > > 524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) >=20 > Wasn't md's blocksize =3D systems's pagesize, IIRC 8k on sparc. > I'm surprised that 1k even works. 1M block size, not 1K :) Kris --i0/AhcQY5QxfSsSZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCe/k/Wry0BWjoQKURAnlpAJ0Ycwo3z9S2edq0eCOqQVDnyTjmDgCgsd7S oLEFW1ev3yYw79F23PP1evw= =sap7 -----END PGP SIGNATURE----- --i0/AhcQY5QxfSsSZ-- From owner-freebsd-smp@FreeBSD.ORG Sat May 7 07:17:26 2005 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ADE6816A4D8; Sat, 7 May 2005 07:17:26 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 96D2543D2F; Sat, 7 May 2005 07:17:25 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j477HPlp014466; Sat, 7 May 2005 09:17:25 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: ticso@cicely.de From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 07 May 2005 00:59:27 +0200." <20050506225926.GB75629@cicely12.cicely.de> Date: Sat, 07 May 2005 09:17:25 +0200 Message-ID: <14465.1115450245@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: smp@freebsd.org cc: current@freebsd.org cc: Kris Kennaway Subject: Re: Benchmarking mpsafevfs with parallel tarball extraction X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 May 2005 07:17:26 -0000 In message <20050506225926.GB75629@cicely12.cicely.de>, Bernd Walter writes: >On Fri, May 06, 2005 at 01:06:35PM -0700, Kris Kennaway wrote: >> On Fri, May 06, 2005 at 11:48:52AM -0700, Kris Kennaway wrote: >> > On Fri, May 06, 2005 at 11:35:29AM -0700, Kris Kennaway wrote: >> > >> > > I might be bumping into the bandwidth of md here - when I ran less >> > > rigorous tests with lower concurrency of extractions I seemed to be >> > > getting marginally better performance (about an effective concurrency >> > > of 2.2 for both 3 and 10 simultaneous extractions - so at least it >> > > doesn't seem to degrade badly). Or this might be reflecting VFS lock >> > > contention (which there is certainly a lot of, according to mutex >> > > profiling traces). >> > >> > I suspect that I am hitting the md bandwidth: >> > >> > # dd if=/dev/zero of=/dev/md0 bs=1024k count=500 >> > 500+0 records in >> > 500+0 records out >> > 524288000 bytes transferred in 9.501760 secs (55177988 bytes/sec) > >Wasn't md's blocksize = systems's pagesize, IIRC 8k on sparc. >I'm surprised that 1k even works. That's only relevant for swap backed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.