From owner-freebsd-performance@FreeBSD.ORG Tue Apr 15 13:08:47 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E44DA106564A for ; Tue, 15 Apr 2008 13:08:47 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id C68DD8FC2D for ; Tue, 15 Apr 2008 13:08:47 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m3FCtmi8091729; Tue, 15 Apr 2008 05:55:52 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m3FCsqW6097238; Tue, 15 Apr 2008 05:54:52 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (61.204.211.246.customerlink.pwd.ne.jp [61.204.211.246]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m3FCspKP023287; Tue, 15 Apr 2008 05:54:51 -0700 (PDT) (envelope-from gnn@neville-neil.com) Date: Tue, 15 Apr 2008 21:54:49 +0900 Message-ID: From: gnn@freebsd.org To: Aristedes Maniatis In-Reply-To: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.11.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Apr 2008 13:08:48 -0000 At Sat, 12 Apr 2008 09:08:13 +1000, Aristedes Maniatis wrote: > > I have a project with 3 workstations all needing high speed access to > about 6Tb of storage. In the past I've installed technology such as > fibre channel connected SAN storage (using Apple's xsan) for up to a > dozen workstations, but with only three workstations for this project > I'm thinking about 10GbE. The workstations will be OSX and the server > FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration. > > Are 10GbE NICs and drivers significantly mature enough under FreeBSD > to accomplish this? I'd need to achieve about 60MB/s transfer rate > which is theoretically quite doable, as long as the drive array can > keep up with three streams of that speed. I'd use netatalk, samba or > nfs to share files depending on which I can eek the best speeds out of. > > Alternatively I could populate the server with 1GbE NICs, one per > workstation and use cross over cable. That way there is absolutely no > contention on the network. > > Any thoughts about the viability of this? Is 10GbE in production use > with FreeBSD and does it scale well? I am working with the Chelsio hardware and it seems to be well supported on FreeBSD. All the machines in that system are FreeBSD though. How do you intend to get the OSX systems to be 10GE? Best, George From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 00:02:35 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 436E6106564A for ; Wed, 16 Apr 2008 00:02:35 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from fish.ish.com.au (eth5921.nsw.adsl.internode.on.net [59.167.240.32]) by mx1.freebsd.org (Postfix) with ESMTP id E75B98FC20 for ; Wed, 16 Apr 2008 00:02:34 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from delish.ish.com.au ([203.29.62.201] helo=ish.com.au) by fish.ish.com.au with esmtps (SSLv3:DES-CBC3-SHA:168) (Exim 4.43) id 1Jlv1s-0006Ua-1j; Wed, 16 Apr 2008 09:57:36 +1000 Received: from ip-182.ish.com.au ([203.29.62.182] verified) by ish.com.au (CommuniGate Pro SMTP 5.2.1) with ESMTP id 3435915; Wed, 16 Apr 2008 10:02:33 +1000 Message-Id: <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> From: Aristedes Maniatis To: gnn@freebsd.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Wed, 16 Apr 2008 10:02:33 +1000 References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-performance@freebsd.org Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 00:02:35 -0000 On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: > I am working with the Chelsio hardware and it seems to be well > supported on FreeBSD. All the machines in that system are FreeBSD > though. How do you intend to get the OSX systems to be 10GE? What sort of throughput are you getting with that setup? Are you using NFS between the systems? Good question about OSX, and I hadn't got to that part yet :-) But I was hoping that some OSX drivers existed. My fail back plan is to put 3 x 1GbE NICs into the server and just use crossover cable between the 3 workstations and the server to avoid any contention within the ethernet network. Ari Maniatis --------------------------> ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 00:09:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4DBB106566B for ; Wed, 16 Apr 2008 00:09:05 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.180]) by mx1.freebsd.org (Postfix) with ESMTP id B7AE68FC20 for ; Wed, 16 Apr 2008 00:09:05 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so3134277waf.3 for ; Tue, 15 Apr 2008 17:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=OVYgUU7WumVCBiFxrYF30DLSEDcZNarn3iTs/419Jxk=; b=idhT/6/our1iRSU1sLK+xuiR7Eipv95B673caIuajLyPtDrFi8wuSpWhrlpkmOo5kl7F/qzY0zi/HQPvRixa2WIeE4+VKDj9YF4o3CHWP7niV9NlqA+YAnNVH7i9SuLywwvj2zkXOpFBVt7Fo/PlqOitDVzMMBh5JwfOf3iV4xk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=L1TYSMExBbT9tp4oTAVQ1/O1tjmvFuZkZw5IbRNnFZmQ3K8j9jeZfbVgvrmYqnLSr3KfOSezaHz6gt7fHbCBIxiCspQWysd5NuZx5V9/feT37l7FbcIT5Fbfvd7TsqtWvObiRHUaYG3f+T3EagpkcuUpa3IUbUK+hlZap+YB+2w= Received: by 10.114.148.2 with SMTP id v2mr5662114wad.173.1208304545197; Tue, 15 Apr 2008 17:09:05 -0700 (PDT) Received: by 10.114.255.16 with HTTP; Tue, 15 Apr 2008 17:09:05 -0700 (PDT) Message-ID: Date: Tue, 15 Apr 2008 17:09:05 -0700 From: "Kip Macy" To: "Aristedes Maniatis" , gnn@freebsd.org, freebsd-performance@freebsd.org In-Reply-To: <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> Cc: Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 00:09:06 -0000 Myricom has OS X support, Chelsio has support in the works. I don't know about Neterion or Intel. -Kip On 4/15/08, Aristedes Maniatis wrote: > > On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: > > I am working with the Chelsio hardware and it seems to be well > > supported on FreeBSD. All the machines in that system are FreeBSD > > though. How do you intend to get the OSX systems to be 10GE? > > > What sort of throughput are you getting with that setup? Are you using > NFS between the systems? > > Good question about OSX, and I hadn't got to that part yet :-) But I > was hoping that some OSX drivers existed. My fail back plan is to put > 3 x 1GbE NICs into the server and just use crossover cable between the > 3 workstations and the server to avoid any contention within the > ethernet network. > > Ari Maniatis > > > --------------------------> > ish > http://www.ish.com.au > Level 1, 30 Wilson Street Newtown 2042 Australia > phone +61 2 9550 5001 fax +61 2 9550 4001 > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 00:17:11 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 572FB106566B for ; Wed, 16 Apr 2008 00:17:11 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from hellskitchen.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.14.82]) by mx1.freebsd.org (Postfix) with ESMTP id 0E5758FC1E for ; Wed, 16 Apr 2008 00:17:10 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY hellskitchen.mr.itd.umich.edu ID 48054577.B7BE5.3598 ; 15 Apr 2008 20:16:56 -0400 Message-ID: <48054577.2050902@umich.edu> Date: Tue, 15 Apr 2008 20:16:55 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: Aristedes Maniatis References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> In-Reply-To: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 00:17:11 -0000 Hi Aristedes, We are/were testing FreeBSD on a Dell PE2950 with a Myricom 10GB PCI-Express copper CX card. The driver seems mature. In tests out of the box, I only saw about 3Gbps from iperf (testing against a linux system...and maybe there are other issues with our environment/that system to tune up yet...i wouldn't take that number too seriously). I think some tuning could get it up to the maximum, anyways. It certainly took a little work to get our Linux systems up to the max, so I would say that in general the defaults with 10G drivers on any system need some work to hit the maximum bandwidth. -Ben Aristedes Maniatis wrote: > I have a project with 3 workstations all needing high speed access to > about 6Tb of storage. In the past I've installed technology such as > fibre channel connected SAN storage (using Apple's xsan) for up to a > dozen workstations, but with only three workstations for this project > I'm thinking about 10GbE. The workstations will be OSX and the server > FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration. > > Are 10GbE NICs and drivers significantly mature enough under FreeBSD to > accomplish this? I'd need to achieve about 60MB/s transfer rate which is > theoretically quite doable, as long as the drive array can keep up with > three streams of that speed. I'd use netatalk, samba or nfs to share > files depending on which I can eek the best speeds out of. > > Alternatively I could populate the server with 1GbE NICs, one per > workstation and use cross over cable. That way there is absolutely no > contention on the network. > > Any thoughts about the viability of this? Is 10GbE in production use > with FreeBSD and does it scale well? > > > Cheers > Ari Maniatis > > > > > --------------------------> > ish > http://www.ish.com.au > Level 1, 30 Wilson Street Newtown 2042 Australia > phone +61 2 9550 5001 fax +61 2 9550 4001 > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > > -- Benjeman Meekhof - UM ATLAS/AGLT2 Computing office: 734-764-3450 cell: 734-417-6312 From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 00:31:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9B5C106566B for ; Wed, 16 Apr 2008 00:31:16 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from tombraider.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.93.161]) by mx1.freebsd.org (Postfix) with ESMTP id 6391A8FC12 for ; Wed, 16 Apr 2008 00:31:16 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY tombraider.mr.itd.umich.edu ID 480548D0.71063.20938 ; 15 Apr 2008 20:31:12 -0400 Message-ID: <480548CF.5080104@umich.edu> Date: Tue, 15 Apr 2008 20:31:11 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 00:31:16 -0000 Hi, I posted earlier about some results with this same system using UFS2. Now trying to test ZFS. This is a Dell PE2950 with two Perc6 controllers and 4 md1000 disk shelves with 750GB drives. 16GB RAM, dual quad core Xeon. I recompiled our kernel to use the ULE scheduler instead of default. I could not get through an entire run of iozone without a system reboot/crash. ZFS is clearly labeled experimental, of course. It seems to die for sure around 10 processes, sometimes less (this is the end of my output from iozone): Children see throughput for 10 readers = 135931.72 KB/sec Parent sees throughput for 10 readers = 135927.24 KB/sec Min throughput per process = 13351.26 KB/sec Max throughput per process = 14172.05 KB/sec Avg throughput per process = 13593.17 KB/sec Min xfer = 31586816.00 KB Some zpool info below - each volume below is a raid6 of 30PD on one controller. I may try different hardware volume configs for fun. zpool create test mfid0 mfid2 # pool is automatically mounted at /test # pool: test # state: ONLINE # scrub: none requested #config: # # NAME STATE READ WRITE CKSUM # test ONLINE 0 0 0 # mfid0 ONLINE 0 0 0 # mfid2 ONLINE 0 0 0 # #errors: No known data errors -Ben From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 05:07:08 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C02541065675 for ; Wed, 16 Apr 2008 05:07:08 +0000 (UTC) (envelope-from swmspam@swmoore.net) Received: from smtpauth01.prod.mesa1.secureserver.net (smtpauth01.prod.mesa1.secureserver.net [64.202.165.181]) by mx1.freebsd.org (Postfix) with SMTP id 9B3C38FC2C for ; Wed, 16 Apr 2008 05:07:08 +0000 (UTC) (envelope-from swmspam@swmoore.net) Received: (qmail 29650 invoked from network); 16 Apr 2008 04:40:26 -0000 Received: from unknown (208.46.203.195) by smtpauth01.prod.mesa1.secureserver.net (64.202.165.181) with ESMTP; 16 Apr 2008 04:40:24 -0000 From: "Stephen Moore" To: Date: Wed, 16 Apr 2008 00:40:20 -0400 Message-ID: <001101c89f7b$fc626600$c3cb2ed0@d400> MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook 11 Thread-index: Acife/mRoaT2wSeHTla/KLeOFlflCA== X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Tweaking Disk Cache/Buffers and fsync X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 05:07:08 -0000 There's a lot of discussion on the sysctl settings for vfs disk cache/buffers, but very little consolidated or comprehensive explanations. I have a NAS box with 1GB of memory, but rarely see over ~40MB of utilization during file transfers. When writing a large file to the NAS, I see the transfer rate throttle and pause in regular intervals. This indicates to me the dirty buffers are flushing and writing to the disk, and the file transfer is interrupted during the disk write process. Therefore, I would like to (1) increase the memory utilization, so that a large file write (500MB) will buffer entirely to memory for fastest transfer speeds, and (2) then take as long as needed to write the dirty buffers to the disk, after the transfer is concluded. For part (1), I looked at the following sysctl settings, but I don't fully understand the interactions: vfs.maxbufspace vfs.lobufspace vfs.hibufspace vfs.hidirtybuffers vfs.hirunningspace also /boot/loader.conf kern.nbuf I understand some rules: hirunningspace should be between 1MB to 4MB. lobufspace should be 25% to 75% of hibufspace. What I don't understand is the relationship between maxbufspace and hibufspace, and what are good values. I also don't understand hidirtybuffers. For part (2), I need to slow down the occurrence of sync() or fsync(). Ideally, a sync would not occur for 300 seconds, which is about 10 times as long as (apparently) the default setting. (obviously, if the buffer space is full, a sync would need to occur to make more room.) I've seen references of a daemon to periodically run a sync, or is it controlled by init? Yes, this increases vulnerability when the data is in buffer before committed to disk, but in my application, this is reasonable. From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 07:31:06 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C764106566B for ; Wed, 16 Apr 2008 07:31:06 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id E3F5A8FC23 for ; Wed, 16 Apr 2008 07:31:05 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m3G7V0i6022029; Wed, 16 Apr 2008 00:31:05 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m3G7UPY0025461; Wed, 16 Apr 2008 00:30:25 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (61.204.211.246.customerlink.pwd.ne.jp [61.204.211.246]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m3G7UOY8078265; Wed, 16 Apr 2008 00:30:25 -0700 (PDT) (envelope-from gnn@neville-neil.com) Date: Wed, 16 Apr 2008 16:30:23 +0900 Message-ID: From: gnn@freebsd.org To: Aristedes Maniatis In-Reply-To: <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.11.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 07:31:06 -0000 At Wed, 16 Apr 2008 10:02:33 +1000, Aristedes Maniatis wrote: > > > On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: > > I am working with the Chelsio hardware and it seems to be well > > supported on FreeBSD. All the machines in that system are FreeBSD > > though. How do you intend to get the OSX systems to be 10GE? > > > What sort of throughput are you getting with that setup? Are you using > NFS between the systems? I haven't done any performance testing of bandwidth as yet and we are not using these for NFS. > Good question about OSX, and I hadn't got to that part yet :-) But I > was hoping that some OSX drivers existed. My fail back plan is to > put 3 x 1GbE NICs into the server and just use crossover cable > between the 3 workstations and the server to avoid any contention > within the ethernet network. I think you want either a pure 10GbE network or a pure 1GbE network because my understanding is that 10GbE switches have issues with mixed links. I have not tested that first hand yet though. Best, George From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 07:43:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8383D1065671 for ; Wed, 16 Apr 2008 07:43:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 3648F8FC1F for ; Wed, 16 Apr 2008 07:43:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 794BA1B10F17; Wed, 16 Apr 2008 09:43:22 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_21 autolearn=no version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 30A021B10F00; Wed, 16 Apr 2008 09:43:10 +0200 (CEST) Message-ID: <4805AE0D.1050005@moneybookers.com> Date: Wed, 16 Apr 2008 10:43:09 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.12 (X11/20080326) MIME-Version: 1.0 To: "Benjeman J. Meekhof" References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <48054577.2050902@umich.edu> In-Reply-To: <48054577.2050902@umich.edu> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/6793/Wed Apr 16 05:57:30 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, Aristedes Maniatis Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 07:43:24 -0000 Greetings, Benjeman J. Meekhof wrote: > Hi Aristedes, > > We are/were testing FreeBSD on a Dell PE2950 with a Myricom 10GB > PCI-Express copper CX card. The driver seems mature. In tests out of > the box, I only saw about 3Gbps from iperf (testing against a linux > system...and maybe there are other issues with our environment/that > system to tune up yet...i wouldn't take that number too seriously). If I remember correctly there is a problem with iperf, because it utilize too much CPU. I saw patches flying around, and part of them are in FreeBSD ports collection I think. Personally I prefer netperf for doing network tests :) > > I think some tuning could get it up to the maximum, anyways. It > certainly took a little work to get our Linux systems up to the max, > so I would say that in general the defaults with 10G drivers on any > system need some work to hit the maximum bandwidth. > > -Ben > > Aristedes Maniatis wrote: >> I have a project with 3 workstations all needing high speed access to >> about 6Tb of storage. In the past I've installed technology such as >> fibre channel connected SAN storage (using Apple's xsan) for up to a >> dozen workstations, but with only three workstations for this project >> I'm thinking about 10GbE. The workstations will be OSX and the server >> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration. >> >> Are 10GbE NICs and drivers significantly mature enough under FreeBSD >> to accomplish this? I'd need to achieve about 60MB/s transfer rate >> which is theoretically quite doable, as long as the drive array can >> keep up with three streams of that speed. I'd use netatalk, samba or >> nfs to share files depending on which I can eek the best speeds out of. >> >> Alternatively I could populate the server with 1GbE NICs, one per >> workstation and use cross over cable. That way there is absolutely no >> contention on the network. >> >> Any thoughts about the viability of this? Is 10GbE in production use >> with FreeBSD and does it scale well? >> >> >> Cheers >> Ari Maniatis >> >> >> >> >> --------------------------> >> ish >> http://www.ish.com.au >> Level 1, 30 Wilson Street Newtown 2042 Australia >> phone +61 2 9550 5001 fax +61 2 9550 4001 >> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >> >> >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to >> "freebsd-performance-unsubscribe@freebsd.org" >> >> > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 08:57:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B630106566C for ; Wed, 16 Apr 2008 08:57:50 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id ABB838FC16 for ; Wed, 16 Apr 2008 08:57:49 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Jm3SZ-0000sa-RI for freebsd-performance@freebsd.org; Wed, 16 Apr 2008 08:57:43 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 08:57:43 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 08:57:43 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 16 Apr 2008 10:57:34 +0200 Lines: 53 Message-ID: References: <480548CF.5080104@umich.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig78DB0A5F8C64169284779A1A" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.12 (X11/20080227) In-Reply-To: <480548CF.5080104@umich.edu> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 08:57:50 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig78DB0A5F8C64169284779A1A Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Benjeman J. Meekhof wrote: > Hi, >=20 > I posted earlier about some results with this same system using UFS2. > Now trying to test ZFS. This is a Dell PE2950 with two Perc6 > controllers and 4 md1000 disk shelves with 750GB drives. 16GB RAM, dua= l > quad core Xeon. I recompiled our kernel to use the ULE scheduler instea= d > of default. >=20 > I could not get through an entire run of iozone without a system > reboot/crash. ZFS is clearly labeled experimental, of course. >=20 > It seems to die for sure around 10 processes, sometimes less (this is > the end of my output from iozone): >=20 > Children see throughput for 10 readers =3D 135931.72 KB/sec > Parent sees throughput for 10 readers =3D 135927.24 = KB/sec > Min throughput per process =3D 13351.26 = KB/sec > Max throughput per process =3D 14172.05 = KB/sec > Avg throughput per process =3D 13593.17 = KB/sec > Min xfer =3D 31586816.00= KB Can you tell us how does this compare to UFS2 results you posted previously? (since you used dd for UFS2 and now iozone for ZFS; what are your conclusions?) --------------enig78DB0A5F8C64169284779A1A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIBb9+ldnAQVacBcgRAvuGAKDV7koR1nUT1qGQuJPspIJ7TYAn3gCglWFY biwr0YQtVlbXPKXkvzEs/dA= =BB1k -----END PGP SIGNATURE----- --------------enig78DB0A5F8C64169284779A1A-- From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 09:59:45 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 758) id A731A1065675; Wed, 16 Apr 2008 09:59:45 +0000 (UTC) Date: Wed, 16 Apr 2008 09:59:45 +0000 From: Kris Kennaway To: Aristedes Maniatis Message-ID: <20080416095945.GA91566@hub.freebsd.org> References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> User-Agent: Mutt/1.4.2.1i Cc: gnn@freebsd.org, freebsd-performance@freebsd.org Subject: Re: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 09:59:45 -0000 On Wed, Apr 16, 2008 at 10:02:33AM +1000, Aristedes Maniatis wrote: > > On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: > >I am working with the Chelsio hardware and it seems to be well > >supported on FreeBSD. All the machines in that system are FreeBSD > >though. How do you intend to get the OSX systems to be 10GE? > > > What sort of throughput are you getting with that setup? Are you using > NFS between the systems? I get about 150 MB/sec NFS random write throughput between chelsio NICs. We are still in the process of optimizing NFS at the high end. For bulk packet throughput it is not difficult to saturate it. Kris -- In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 15:17:56 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3243106566B; Wed, 16 Apr 2008 15:17:55 +0000 (UTC) (envelope-from Leonid.Grossman@neterion.com) Received: from owa.neterion.com (mx.neterion.com [72.1.205.142]) by mx1.freebsd.org (Postfix) with ESMTP id BBB6A8FC16; Wed, 16 Apr 2008 15:17:55 +0000 (UTC) (envelope-from Leonid.Grossman@neterion.com) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Wed, 16 Apr 2008 11:05:54 -0400 Message-ID: <78C9135A3D2ECE4B8162EBDCE82CAD770367B106@nekter> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 10GbE speeds Thread-Index: AcifVjCj0RlMzWEXSGqzy5zDNaZarwAe691g References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au><15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> From: "Leonid Grossman" To: "Kip Macy" , "Aristedes Maniatis" , , X-Mailman-Approved-At: Wed, 16 Apr 2008 15:32:15 +0000 Cc: Subject: RE: 10GbE speeds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 15:17:56 -0000 Neterion supports OS X on 10GbE NICs, I think Intel does as well (via third party). Leonid > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd- > performance@freebsd.org] On Behalf Of Kip Macy > Sent: Tuesday, April 15, 2008 5:09 PM > To: Aristedes Maniatis; gnn@freebsd.org; freebsd-performance@freebsd.org > Subject: Re: 10GbE speeds >=20 > Myricom has OS X support, Chelsio has support in the works. I don't > know about Neterion or Intel. >=20 > -Kip >=20 >=20 >=20 > On 4/15/08, Aristedes Maniatis wrote: > > > > On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: > > > I am working with the Chelsio hardware and it seems to be well > > > supported on FreeBSD. All the machines in that system are FreeBSD > > > though. How do you intend to get the OSX systems to be 10GE? > > > > > > What sort of throughput are you getting with that setup? Are you using > > NFS between the systems? > > > > Good question about OSX, and I hadn't got to that part yet :-) But I > > was hoping that some OSX drivers existed. My fail back plan is to put > > 3 x 1GbE NICs into the server and just use crossover cable between the > > 3 workstations and the server to avoid any contention within the > > ethernet network. > > > > Ari Maniatis > > > > > > --------------------------> > > ish > > http://www.ish.com.au > > Level 1, 30 Wilson Street Newtown 2042 Australia > > phone +61 2 9550 5001 fax +61 2 9550 4001 > > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A > > > > > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to > > "freebsd-performance-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance- > unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 15:58:32 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4458C106564A; Wed, 16 Apr 2008 15:58:32 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from hackers.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.14.81]) by mx1.freebsd.org (Postfix) with ESMTP id E41A28FC29; Wed, 16 Apr 2008 15:58:31 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM [141.211.98.120] (Unknown [141.211.98.120]) BY hackers.mr.itd.umich.edu ID 48062223.66714.2281 ; 16 Apr 2008 11:58:27 -0400 Message-ID: <4806221C.20202@umich.edu> Date: Wed, 16 Apr 2008 11:58:20 -0400 From: Benjeman Meekhof User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: Ivan Voras References: <480548CF.5080104@umich.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 15:58:32 -0000 Sure, here is an example iozone output when I tested UFS2. Same hardware config as with ZFS test. #gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 #newfs -U -b 65536 /dev/stripe/test "Throughput report Y-axis is type of test X-axis is number of processes" "Record size = 512 Kbytes " "Output is in Kbytes/sec" " Initial write " 576953.38 613457.19 627091.58 626818.95 641763.54 626005.11 579073.92 553088.47 557498.71 556188.95 553340.92 548801.62 " Rewrite " 580755.31 621562.75 638209.55 573171.69 633114.27 616501.92 542520.70 541662.21 525603.64 529194.37 506230.25 490589.89 " Read " 505978.28 546837.12 565786.34 310994.23 309813.64 329930.91 351162.15 376940.64 408561.11 432157.69 452106.75 470176.39 " Re-read " 523917.72 581796.50 592393.28 314724.70 308485.12 327409.40 350913.96 381370.12 408105.25 434168.93 458742.49 475407.44 -Ben Ivan Voras wrote: > Benjeman J. Meekhof wrote: >> Hi, >> >> I posted earlier about some results with this same system using UFS2. >> Now trying to test ZFS. This is a Dell PE2950 with two Perc6 >> controllers and 4 md1000 disk shelves with 750GB drives. 16GB RAM, dual >> quad core Xeon. I recompiled our kernel to use the ULE scheduler instead >> of default. >> >> I could not get through an entire run of iozone without a system >> reboot/crash. ZFS is clearly labeled experimental, of course. >> >> It seems to die for sure around 10 processes, sometimes less (this is >> the end of my output from iozone): >> >> Children see throughput for 10 readers = 135931.72 KB/sec >> Parent sees throughput for 10 readers = 135927.24 KB/sec >> Min throughput per process = 13351.26 KB/sec >> Max throughput per process = 14172.05 KB/sec >> Avg throughput per process = 13593.17 KB/sec >> Min xfer = 31586816.00 KB > > Can you tell us how does this compare to UFS2 results you posted > previously? (since you used dd for UFS2 and now iozone for ZFS; what are > your conclusions?) > From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 20:21:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44755106564A for ; Wed, 16 Apr 2008 20:21:16 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id C3C338FC16 for ; Wed, 16 Apr 2008 20:21:15 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JmE7y-0002OS-1z for freebsd-performance@freebsd.org; Wed, 16 Apr 2008 20:21:10 +0000 Received: from 78-0-75-212.adsl.net.t-com.hr ([78.0.75.212]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 20:21:10 +0000 Received: from ivoras by 78-0-75-212.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 20:21:10 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 16 Apr 2008 22:20:58 +0200 Lines: 104 Message-ID: References: <480548CF.5080104@umich.edu> <4806221C.20202@umich.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBA51353FEBBD3C8D1F09D96B" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-75-212.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) In-Reply-To: <4806221C.20202@umich.edu> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 20:21:16 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBA51353FEBBD3C8D1F09D96B Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Benjeman Meekhof wrote: > Sure, here is an example iozone output when I tested UFS2. Same=20 > hardware config as with ZFS test. >=20 > #gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 > #newfs -U -b 65536 /dev/stripe/test >=20 >=20 > "Throughput report Y-axis is type of test X-axis is number of processes= " > "Record size =3D 512 Kbytes " > "Output is in Kbytes/sec" >=20 > " Read " 505978.28 546837.12 565786.34 310994.23=20 > 309813.64 329930.91 351162.15 376940.64 408561.11 432157.69=20 > 452106.75 470176.39 >=20 > " Re-read " 523917.72 581796.50 592393.28 314724.70=20 > 308485.12 327409.40 350913.96 381370.12 408105.25 434168.93=20 > 458742.49 475407.44 This is hard to compare to what you've posted before, but if it means=20 that with ZFS you get 136 MB/s and with UFS 300-593 MB/s, something is=20 wrong. Per my discussion with Scott Long Can you repeat the test for UFS, but=20 create gstripe with a really small stripe size, like 4 KB? > -Ben >=20 > Ivan Voras wrote: >> Benjeman J. Meekhof wrote: >>> Hi, >>> >>> I posted earlier about some results with this same system using UFS2.= >>> Now trying to test ZFS. This is a Dell PE2950 with two Perc6 >>> controllers and 4 md1000 disk shelves with 750GB drives. 16GB RAM, d= ual >>> quad core Xeon. I recompiled our kernel to use the ULE scheduler inst= ead >>> of default. >>> >>> I could not get through an entire run of iozone without a system >>> reboot/crash. ZFS is clearly labeled experimental, of course. >>> >>> It seems to die for sure around 10 processes, sometimes less (this is= >>> the end of my output from iozone): >>> >>> Children see throughput for 10 readers =3D 135931.72 KB/se= c >>> Parent sees throughput for 10 readers =3D 135927.2= 4=20 >>> KB/sec >>> Min throughput per process =3D 13351.2= 6=20 >>> KB/sec >>> Max throughput per process =3D 14172.0= 5=20 >>> KB/sec >>> Avg throughput per process =3D 13593.1= 7=20 >>> KB/sec >>> Min xfer =3D 31586816.= 00 KB >> >> Can you tell us how does this compare to UFS2 results you posted >> previously? (since you used dd for UFS2 and now iozone for ZFS; what a= re >> your conclusions?) >> >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to=20 > "freebsd-performance-unsubscribe@freebsd.org" >=20 --------------enigBA51353FEBBD3C8D1F09D96B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIBl+qldnAQVacBcgRAkD9AKCQhi9EhJDYGWjjUXtn049XL5KJuQCgh+xZ eCSZf8GKbzZOE2gSJJMXaNo= =+qTO -----END PGP SIGNATURE----- --------------enigBA51353FEBBD3C8D1F09D96B-- From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 20:38:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4F78106566B for ; Wed, 16 Apr 2008 20:38:16 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7BE528FC20 for ; Wed, 16 Apr 2008 20:38:16 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JmEOV-0003ry-OG for freebsd-performance@freebsd.org; Wed, 16 Apr 2008 20:38:15 +0000 Received: from 78-0-75-212.adsl.net.t-com.hr ([78.0.75.212]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 20:38:15 +0000 Received: from ivoras by 78-0-75-212.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Apr 2008 20:38:15 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 16 Apr 2008 22:38:04 +0200 Lines: 30 Message-ID: References: <480548CF.5080104@umich.edu> <4806221C.20202@umich.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA1AB7BEFD882C9D6F5F2628F" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-0-75-212.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) In-Reply-To: X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 20:38:16 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA1AB7BEFD882C9D6F5F2628F Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Ivan Voras wrote: > Per my discussion with Scott Long Can you repeat the test for UFS, but = > create gstripe with a really small stripe size, like 4 KB? Actually, no need to do that - it looks like iozone is doing quite=20 random IO ops so it won't help you. --------------enigA1AB7BEFD882C9D6F5F2628F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIBmOsldnAQVacBcgRAtgXAKDS+/R1hKzOTNvhpgkKKTm3qWS7WwCgmr7H OKewklUHM1Ez4yezyCFsKlI= =difp -----END PGP SIGNATURE----- --------------enigA1AB7BEFD882C9D6F5F2628F-- From owner-freebsd-performance@FreeBSD.ORG Wed Apr 16 21:35:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3D25106566B for ; Wed, 16 Apr 2008 21:35:59 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from tombraider.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.93.161]) by mx1.freebsd.org (Postfix) with ESMTP id 5BE8D8FC20 for ; Wed, 16 Apr 2008 21:35:58 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY tombraider.mr.itd.umich.edu ID 4806713B.DCA64.29291 ; 16 Apr 2008 17:35:56 -0400 Message-ID: <4806713B.2060405@umich.edu> Date: Wed, 16 Apr 2008 17:35:55 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <480548CF.5080104@umich.edu> <4806221C.20202@umich.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: ZFS, Dell PE2950 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Apr 2008 21:35:59 -0000 I have some old dd numbers from when I was experimenting to find a UFS/gstripe combination that wasn't horrifyingly slow to read. I was not then adjusting filesystem blocksize, and not until moving UFS2 bs to the maximum did initial results seem worth resuming iozone tests. Raid HW stripe-width is 128k. FWIW. # using 4k stripe, same as above #gstripe label -v -s 4k test /dev/mfid0 /dev/mfid2 #newfs -U /dev/stripe/test #time dd if=/dev/zero of=/test/deletafile bs=1M count=10240 #time dd if=/test/deletafile of=/dev/null bs=1M count=10240 #write: 26.5s 403665800 bps #read: 157s 68343843 bps -Ben Ivan Voras wrote: > Ivan Voras wrote: > >> Per my discussion with Scott Long Can you repeat the test for UFS, but >> create gstripe with a really small stripe size, like 4 KB? > > Actually, no need to do that - it looks like iozone is doing quite > random IO ops so it won't help you. > From owner-freebsd-performance@FreeBSD.ORG Thu Apr 17 09:01:46 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53D64106566B for ; Thu, 17 Apr 2008 09:01:46 +0000 (UTC) (envelope-from mk@mkdev.eu) Received: from natrium.sulf.at (natrium.sulf.at [88.198.116.108]) by mx1.freebsd.org (Postfix) with ESMTP id C34428FC20 for ; Thu, 17 Apr 2008 09:01:45 +0000 (UTC) (envelope-from mk@mkdev.eu) Received: from [192.168.101.179] (dslb-084-057-026-112.pools.arcor-ip.net [84.57.26.112]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by natrium.sulf.at (Postfix) with ESMTP id 43168114B7 for ; Thu, 17 Apr 2008 08:52:06 +0000 (UTC) Message-ID: <48070E93.9030705@mkdev.eu> Date: Thu, 17 Apr 2008 10:47:15 +0200 From: Markus Klaschka Organization: mkdev.eu User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.8.1.11) Gecko/20080208 SeaMonkey/1.1.7 Mnenhy/0.7.5.666 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> <20080416095945.GA91566@hub.freebsd.org> In-Reply-To: <20080416095945.GA91566@hub.freebsd.org> Content-Type: multipart/mixed; boundary="------------000408040305000101090504" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: NFS performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2008 09:01:46 -0000 This is a multi-part message in MIME format. --------------000408040305000101090504 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit That's interesting cause heavy reading from NFS brought me a loadavg of 70 and more if there were a lot of small files to read. I thought this is a normal issue about NFS... By the way, are all Realtek Cards for the bin or only the 8139...the server has a 'RTL8169 Gigabit Ethernet Adapter' I mean, you all know this, if not read the comments in that file ;) root@kalium:~ > grep worst /usr/src/sys/pci/if_rl.c * probably the worst PCI ethernet controller ever made, with the possible What could be configured wrong? What's the best way to test bandwidth if I only got one well connected server? pathchar? Cheers Markus Kris Kennaway schrieb: > On Wed, Apr 16, 2008 at 10:02:33AM +1000, Aristedes Maniatis wrote: > >> On 15/04/2008, at 10:54 PM, gnn@freebsd.org wrote: >> >>> I am working with the Chelsio hardware and it seems to be well >>> supported on FreeBSD. All the machines in that system are FreeBSD >>> though. How do you intend to get the OSX systems to be 10GE? >>> >> What sort of throughput are you getting with that setup? Are you using >> NFS between the systems? >> > > I get about 150 MB/sec NFS random write throughput between chelsio > NICs. We are still in the process of optimizing NFS at the high end. > For bulk packet throughput it is not difficult to saturate it. > > Kris > > -- > In God we Trust -- all others must submit an X.509 certificate. > -- Charles Forsythe > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -- Markus Klaschka MKDev - Markus Klaschka Development http://www.mkdev.eu Spain: 0034 - 63 747 23 07 UK: 0044 - 750 910 2718 Mail: mk@mkdev.eu Skype: mark-use IRC: mark-use @ irc.freenode.net : #freebsd, ##security, #freebsd-src, #bsdforen.de, #bsdgroup.de --------------000408040305000101090504-- From owner-freebsd-performance@FreeBSD.ORG Thu Apr 17 12:31:42 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: by hub.freebsd.org (Postfix, from userid 758) id 3D9501065670; Thu, 17 Apr 2008 12:31:42 +0000 (UTC) Date: Thu, 17 Apr 2008 12:31:42 +0000 From: Kris Kennaway To: Markus Klaschka Message-ID: <20080417123142.GI25623@hub.freebsd.org> References: <5873E91C-C096-4EE1-A5F5-4BCE110E2EE7@ish.com.au> <15A6FBF6-052E-486D-9470-CAE5819BE93F@ish.com.au> <20080416095945.GA91566@hub.freebsd.org> <48070E93.9030705@mkdev.eu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48070E93.9030705@mkdev.eu> User-Agent: Mutt/1.4.2.1i Cc: freebsd-performance@freebsd.org Subject: Re: NFS performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2008 12:31:42 -0000 On Thu, Apr 17, 2008 at 10:47:15AM +0200, Markus Klaschka wrote: > That's interesting cause heavy reading from NFS brought me a loadavg of > 70 and more if there were a lot of small files to read. > I thought this is a normal issue about NFS... > By the way, are all Realtek Cards for the bin or only the 8139...the > server has a 'RTL8169 Gigabit Ethernet Adapter' > I mean, you all know this, if not read the comments in that file ;) > > root@kalium:~ > grep worst /usr/src/sys/pci/if_rl.c > * probably the worst PCI ethernet controller ever made, with the > possible > > What could be configured wrong? > What's the best way to test bandwidth if I only got one well connected > server? pathchar? NFS performance is limited by several things: * server disk I/O. With low end disk hardware you are not going to get good performance at high load. * network bandwidth. Ditto. * NFS client and server implementation There have been important relevant improvements in 8.0 that improve the client performance with many concurrent processes doing NFS I/O. Also 7.0 has much better performance than 6.x. Kris P.S. Load average doesn't tell you if your system is performing badly, it tells you that the system is running many processes. -- In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe