From owner-freebsd-performance@FreeBSD.ORG Mon Apr 25 10:48:09 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCFFD16A4CE for ; Mon, 25 Apr 2005 10:48:09 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3565A43D3F for ; Mon, 25 Apr 2005 10:48:09 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 9B1AC46B89 for ; Mon, 25 Apr 2005 06:48:08 -0400 (EDT) Date: Mon, 25 Apr 2005 11:50:14 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: performance@FreeBSD.org In-Reply-To: <20050417134448.L85588@fledge.watson.org> Message-ID: <20050425114546.O74930@fledge.watson.org> References: <20050417134448.L85588@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: Memory allocation performance/statistics patches X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2005 10:48:09 -0000 I now have updated versions of these patches, which correct some inconsistencies in approach (universal use of curcpu now, for example), remove some debugging code, etc. I've received relatively little performance feedback on them, and would appreciate it if I could get some. :-) Especially as to whether these impact disk I/O related workloads, useful macrobenchmarks, etc. The latest patch is at: http://www.watson.org/~robert/freebsd/netperf/20050425-uma-mbuf-malloc-critical.diff The changes in the following files in the combined patch are intended to be broken out in to separate patches, as desired, as follows: kern_malloc.c malloc.diff kern_mbuf.c mbuf.diff uipc_mbuf.c mbuf.diff uipc_syscalls.c mbuf.diff malloc.h malloc.diff mbuf.h mbuf.diff pcpu.h malloc.diff, mbuf.diff, uma.diff uma_core.c uma.diff uma_int.h uma.diff I.e., the pcpu.h changes are a dependency for all of the remaining changes. As before, I'm interested in both the impact of individual patches, and the net effect of the total change associated with all patches applied. Because this diff was generated by p4, patch may need some help in identifying the targets of each part of the diff. Robert N M Watson On Sun, 17 Apr 2005, Robert Watson wrote: > > Attached please find three patches: > > (1) uma.diff, which modifies the UMA slab allocator to use critical > sections instead of mutexes to protect per-CPU caches. > > (2) malloc.diff, which modifies the malloc memory allocator to use > critical sections and per-CPU data instead of mutexes to store > per-malloc-type statistics, coalescing for the purposes of the sysctl > used to generate vmstat -m output. > > (3) mbuf.diff, which modifies the mbuf allocator to use per-CPU data and > critical sections for statistics, instead of synchronization-free > statistics which could result in substantial inconsistency on SMP > systems. > > These changes are facilitated by John Baldwin's recent re-introduction of > critical section optimizations that permit critical sections to be > implemented "in software", rather than using the hardware interrupt disable > mechanism, which is quite expensive on modern processors (especially Xeon P4 > CPUs). While not identical, this is similar to the softspl behavior in 4.x, > and Linux's preemption disable mechanisms (and various other post-Vax systems > :-)). > > The reason this is interesting is that it allows synchronization of per-CPU > data to be performed at a much lower cost than previously, and consistently > across UP and SMP systems. Prior to these changes, the use of critical > sections and per-CPU data as an alternative to mutexes would lead to an > improvement on SMP, but not on UP. So, that said, here's what I'd like us to > look at: > > - Patches (1) and (2) are intended to improve performance by reducing the > overhead of maintaining cache consistency and statistics for UMA and > malloc(9), and may universally impact performance (in a small way) due > to the breadth of their use through the kernel. > > - Patch (3) is intended to restore consistency to statistics in the > presence of SMP and preemption, at the possible cost of some > performance. > > I'd like to confirm that for the first two patches, for interesting > workloads, performance generally improves, and that stability doesn't > degrade. For the third partch, I'd like to quantify the cost of the changes > for interesting workloads, and likewise confirm no loss of stability. > > Because these will have a relatively small impact, a fair amount of caution > is required in testing. We may be talking about a percent or two, maybe > four, difference in benchmark performance, and many benchmarks have a higher > variance than that. > > A couple of observations for those interested: > > - The INVARIANTS panic with UMA seen in some earlier patch versions is > believed to be corrected. > > - Right now, because I use arrays of foo[MAXCPUS], I'm concerned that > different CPUs will be writing to the same cache line as they're > adjacent in memory. Moving to per-CPU chunks of memory to hold this > stuff is desirable, but I think first we need to identify a model by > which to do that cleanly. I'm not currently enamored of the 'struct > pcpu' model, since it makes us very sensitive to ABI changes, as well as > not offering a model by which modules can register new per-cpu data > cleanly. I'm also inconsistent about how I dereference into the arrays, > and intend to move to using 'curcpu' throughout. > > - Because mutexes are no longer used in UMA, and not for the others > either, stats read across different CPUs that are coalesced may be > slightly inconsistent. I'm not all that concerned about it, but it's > worth thinking on. > > - Malloc stats for realloc() are still broken if you apply this patch. > > - High watermarks are no longer maintained for malloc since they require a > global notion of "high" that is tracked continuously (i.e., at each > change), and there's no longer a global view except when the observer > kicks in (sysctl). You can imagine various models to restore some > notion of a high watermark, but I'm not currently sure which is the > best. The high watermark notion is desirable though. > > So this is a request for: > > (1) Stability testing of these patches. Put them on a machine, make them > hurt. If things go South, try applying the patches one by one until > it's clear which is the source. > > (2) Performance testing of these patches. Subject to the challenges in > testing them. If you are interested, please test each patch > separately to evaluate its impact on your system. Then apply all > together and see how it evens out. You may find that the mbuf > allocator patch outweighs the benefits of the other two patches, if > so, that is interesting and something to work on! > > I've done some micro-benchmarking using tools like netblast, syscall_timing, > etc, but I'm interested particularly in the impact on macrobenchmarks. > > Thanks! > > Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Mon Apr 25 17:10:05 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9415416A4CE for ; Mon, 25 Apr 2005 17:10:05 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 071A343D1D for ; Mon, 25 Apr 2005 17:10:05 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 54DF246B03 for ; Mon, 25 Apr 2005 13:10:04 -0400 (EDT) Date: Mon, 25 Apr 2005 18:12:12 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: performance@FreeBSD.org In-Reply-To: <20050425114546.O74930@fledge.watson.org> Message-ID: <20050425181101.Y74930@fledge.watson.org> References: <20050417134448.L85588@fledge.watson.org> <20050425114546.O74930@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: Memory allocation performance/statistics patches X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2005 17:10:05 -0000 On Mon, 25 Apr 2005, Robert Watson wrote: > I now have updated versions of these patches, which correct some > inconsistencies in approach (universal use of curcpu now, for example), > remove some debugging code, etc. I've received relatively little > performance feedback on them, and would appreciate it if I could get > some. :-) Especially as to whether these impact disk I/O related > workloads, useful macrobenchmarks, etc. The latest patch is at: > > > http://www.watson.org/~robert/freebsd/netperf/20050425-uma-mbuf-malloc-critical.diff FYI: For those set up to track perforce, you can find the contents of this patch in: //depot/user/rwatson/percpu/... In addition, that branch also contains diagnostic micro-benchmarks in the kernel to measure the cost of various synchronization operations, memory allocation operations, etc, which can be queried using "sysctl test". Robert N M Watson > > The changes in the following files in the combined patch are intended to be > broken out in to separate patches, as desired, as follows: > > kern_malloc.c malloc.diff > kern_mbuf.c mbuf.diff > uipc_mbuf.c mbuf.diff > uipc_syscalls.c mbuf.diff > malloc.h malloc.diff > mbuf.h mbuf.diff > pcpu.h malloc.diff, mbuf.diff, uma.diff > uma_core.c uma.diff > uma_int.h uma.diff > > I.e., the pcpu.h changes are a dependency for all of the remaining changes. > As before, I'm interested in both the impact of individual patches, and the > net effect of the total change associated with all patches applied. > > Because this diff was generated by p4, patch may need some help in > identifying the targets of each part of the diff. > > Robert N M Watson > > On Sun, 17 Apr 2005, Robert Watson wrote: > >> >> Attached please find three patches: >> >> (1) uma.diff, which modifies the UMA slab allocator to use critical >> sections instead of mutexes to protect per-CPU caches. >> >> (2) malloc.diff, which modifies the malloc memory allocator to use >> critical sections and per-CPU data instead of mutexes to store >> per-malloc-type statistics, coalescing for the purposes of the sysctl >> used to generate vmstat -m output. >> >> (3) mbuf.diff, which modifies the mbuf allocator to use per-CPU data and >> critical sections for statistics, instead of synchronization-free >> statistics which could result in substantial inconsistency on SMP >> systems. >> >> These changes are facilitated by John Baldwin's recent re-introduction of >> critical section optimizations that permit critical sections to be >> implemented "in software", rather than using the hardware interrupt disable >> mechanism, which is quite expensive on modern processors (especially Xeon >> P4 CPUs). While not identical, this is similar to the softspl behavior in >> 4.x, and Linux's preemption disable mechanisms (and various other post-Vax >> systems :-)). >> >> The reason this is interesting is that it allows synchronization of per-CPU >> data to be performed at a much lower cost than previously, and consistently >> across UP and SMP systems. Prior to these changes, the use of critical >> sections and per-CPU data as an alternative to mutexes would lead to an >> improvement on SMP, but not on UP. So, that said, here's what I'd like us >> to look at: >> >> - Patches (1) and (2) are intended to improve performance by reducing the >> overhead of maintaining cache consistency and statistics for UMA and >> malloc(9), and may universally impact performance (in a small way) due >> to the breadth of their use through the kernel. >> >> - Patch (3) is intended to restore consistency to statistics in the >> presence of SMP and preemption, at the possible cost of some >> performance. >> >> I'd like to confirm that for the first two patches, for interesting >> workloads, performance generally improves, and that stability doesn't >> degrade. For the third partch, I'd like to quantify the cost of the >> changes for interesting workloads, and likewise confirm no loss of >> stability. >> >> Because these will have a relatively small impact, a fair amount of caution >> is required in testing. We may be talking about a percent or two, maybe >> four, difference in benchmark performance, and many benchmarks have a >> higher variance than that. >> >> A couple of observations for those interested: >> >> - The INVARIANTS panic with UMA seen in some earlier patch versions is >> believed to be corrected. >> >> - Right now, because I use arrays of foo[MAXCPUS], I'm concerned that >> different CPUs will be writing to the same cache line as they're >> adjacent in memory. Moving to per-CPU chunks of memory to hold this >> stuff is desirable, but I think first we need to identify a model by >> which to do that cleanly. I'm not currently enamored of the 'struct >> pcpu' model, since it makes us very sensitive to ABI changes, as well as >> not offering a model by which modules can register new per-cpu data >> cleanly. I'm also inconsistent about how I dereference into the arrays, >> and intend to move to using 'curcpu' throughout. >> >> - Because mutexes are no longer used in UMA, and not for the others >> either, stats read across different CPUs that are coalesced may be >> slightly inconsistent. I'm not all that concerned about it, but it's >> worth thinking on. >> >> - Malloc stats for realloc() are still broken if you apply this patch. >> >> - High watermarks are no longer maintained for malloc since they require a >> global notion of "high" that is tracked continuously (i.e., at each >> change), and there's no longer a global view except when the observer >> kicks in (sysctl). You can imagine various models to restore some >> notion of a high watermark, but I'm not currently sure which is the >> best. The high watermark notion is desirable though. >> >> So this is a request for: >> >> (1) Stability testing of these patches. Put them on a machine, make them >> hurt. If things go South, try applying the patches one by one until >> it's clear which is the source. >> >> (2) Performance testing of these patches. Subject to the challenges in >> testing them. If you are interested, please test each patch >> separately to evaluate its impact on your system. Then apply all >> together and see how it evens out. You may find that the mbuf >> allocator patch outweighs the benefits of the other two patches, if >> so, that is interesting and something to work on! >> >> I've done some micro-benchmarking using tools like netblast, >> syscall_timing, etc, but I'm interested particularly in the impact on >> macrobenchmarks. >> >> Thanks! >> >> Robert N M Watson > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Tue Apr 26 15:48:05 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B206716A4D4 for ; Tue, 26 Apr 2005 15:48:05 +0000 (GMT) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id D8CE543D39 for ; Tue, 26 Apr 2005 15:48:04 +0000 (GMT) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1DQSAN-0002GS-RX for freebsd-performance@freebsd.org; Tue, 26 Apr 2005 17:40:03 +0200 Received: from 85-64-148-182.barak-online.net ([85.64.148.182]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 26 Apr 2005 17:40:03 +0200 Received: from alexvn by 85-64-148-182.barak-online.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 26 Apr 2005 17:40:03 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: "Alex Vinokur" Date: Tue, 26 Apr 2005 17:35:53 +0300 Lines: 12 Message-ID: X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 85-64-148-182.barak-online.net X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2800.1437 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2800.1441 Sender: news X-Mailman-Approved-At: Wed, 27 Apr 2005 03:41:03 +0000 Subject: C/C++ Program Perfometer X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Apr 2005 15:48:05 -0000 The C/C++ Program Perfometer is an open source tool which enables the programmer to measure the comparative performance of a C/C++ program or of separated pieces of code by one of several desired metrics: e.g., time, memory, or metrics defined by the programmer. http://lists.sourceforge.net/lists/listinfo/cpp-perfometer-users -- Alex Vinokur email: alex DOT vinokur AT gmail DOT com http://mathforum.org/library/view/10978.html http://sourceforge.net/users/alexvn From owner-freebsd-performance@FreeBSD.ORG Wed Apr 27 03:53:18 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 140EF16A4CE for ; Wed, 27 Apr 2005 03:53:18 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4120343D46 for ; Wed, 27 Apr 2005 03:53:17 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001363215.msg for ; Wed, 27 Apr 2005 04:49:10 +0100 Message-ID: <009a01c54adc$89438010$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Sean" , "Eric Anderson" References: <136272710504220832793dfc3d@mail.gmail.com><426920E5.4070806@centtech.com> <13627271050422095030d5dc39@mail.gmail.com> <00a101c5475e$53719de0$7f06000a@int.mediasurface.com> Date: Wed, 27 Apr 2005 04:52:26 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Wed, 27 Apr 2005 04:49:10 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Wed, 27 Apr 2005 04:49:11 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Channel bonding. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2005 03:53:18 -0000 Finally got this machine up 5.4-STABLE Dual 244 2GB RAM 5 x Seagate 400GB on HPT1820A Initial tests using: nttcp -T -n100000 from two clients 1. WinXP P4 Intel gE 2. FreeBSD 5.3-RELEASE dual PIII 733 Intel gE ( 4x WD250 IDE on HPT374) To the test machine 78MB/s I could well be saturating the clients will get another machine hooked up and see if it does more. Using proftpd on the test box and ftp clients on the other two max was 54MB/s dd if=/dev/da0 of=/dev/null bs=128k count=10000 10000+0 records in 10000+0 records out 1310720000 bytes transferred in 24.067986 secs (54459064 bytes/sec) time cat current.iso > /dev/null 0.007u 0.478s 0:00.48 97.9% 10+184k 0+0io 1pf+0w 528Mb/s ( memory speed ) On the old PIII I get: dd if=/dev/da0 of=/dev/null bs=128k count=10000 10000+0 records in 10000+0 records out 1310720000 bytes transferred in 39.415163 secs (33254207 bytes/sec) time cat current.iso > /dev/null 0.054u 1.585s 0:01.64 99.3% 10+180k 0+0io 0pf+0w 159Mb/s ( memory speed ) Comments: dd speed seems very low, netspeed seems good using nttcp considering its using the cheapo onboard BCM5705 Steve ----- Original Message ----- From: "Steven Hartland" killing@multiplay.co.uk >I will be putting together a dual Opteron this weekend with the hope > of testing network throughput. > Spec will be: > Dual 244, 2Gb RAM, 5x400Gb SATA RAID 5 on a Highpoint 1820a > Broadcom 5705, Intel gE and a Intel dual port ( PCI 32 ) for comparison. > > Will let you know the results. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 14:35:44 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DAA4916A4CE; Thu, 28 Apr 2005 14:35:44 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1F38143D49; Thu, 28 Apr 2005 14:35:44 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001366665.msg; Thu, 28 Apr 2005 15:32:14 +0100 Message-ID: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: , Date: Thu, 28 Apr 2005 15:35:39 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 15:32:14 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 15:32:15 +0100 Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 14:35:45 -0000 Sorry wanted to send to performance not current :) Steve ----- Original Message ----- > I've just finished putting together a new server box spec: > Dual AMD 244, 2GB ram, 5 * Seagate SATA 400GB on a > Highpoint 1820a RAID 5 array. > The machine is currently running 5.4-STABLE ( from the > weekend ) After install I did some basic tests and the > disk is return very poor performance low in fact than a > single disk on a bog standard ATA 100 controller: > > 5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 13.348032 secs (49097875 bytes/sec) > > 5.3-RELEASE Highpoint 454 RAID 5 ( 4 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 20.410034 secs (32109697 bytes/sec) > > 5.2.1-RELEASE Intel ICH3 UDMA100 ( 1 disk ) > dd if=/dev/ad0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 11.142405 secs (58816745 bytes/sec) > > Obviously something is seriously a miss here somewhere as > both the RAID 5 arrays a producing less throughput than > a single disk. > > Where do I start looking? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 14:43:13 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A671A16A4CE for ; Thu, 28 Apr 2005 14:43:13 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF59E43D1D for ; Thu, 28 Apr 2005 14:43:12 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j3SEhBJS064541; Thu, 28 Apr 2005 09:43:11 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <4270F648.5030501@centtech.com> Date: Thu, 28 Apr 2005 09:42:16 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <069901c54bfd$2967ba40$7f06000a@int.mediasurface.com> In-Reply-To: <069901c54bfd$2967ba40$7f06000a@int.mediasurface.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/857/Thu Apr 28 01:30:10 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 14:43:13 -0000 Steven Hartland wrote: > I've just finished putting together a new server box spec: > Dual AMD 244, 2GB ram, 5 * Seagate SATA 400GB on a > Highpoint 1820a RAID 5 array. > The machine is currently running 5.4-STABLE ( from the > weekend ) After install I did some basic tests and the > disk is return very poor performance low in fact than a > single disk on a bog standard ATA 100 controller: > > 5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 13.348032 secs (49097875 bytes/sec) > > 5.3-RELEASE Highpoint 454 RAID 5 ( 4 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 20.410034 secs (32109697 bytes/sec) > > 5.2.1-RELEASE Intel ICH3 UDMA100 ( 1 disk ) > dd if=/dev/ad0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 11.142405 secs (58816745 bytes/sec) > > Obviously something is seriously a miss here somewhere as > both the RAID 5 arrays a producing less throughput than > a single disk. > > Where do I start looking? First, understand that RAID 5 is dependant on fast hardware to performa the XOR operations. A single disk without any RAID can easily outperform a RAID array if the RAID array is on a 'slow' controller. The Highpoint controllers are not exactly high-end fast RAID controllers, so ~50MB/s isn't too bad for that I would say. Did you happen to try a vinum RAID5? How about a stripe? You could make two RAID 5's with the controller, and then stripe those in vinum. You could also do a combo of two stripes in the controller, and a vinum mirror in FreeBSD - that would give you decent performance. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 14:58:26 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7E45216A4CF for ; Thu, 28 Apr 2005 14:58:26 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id D522D43D41 for ; Thu, 28 Apr 2005 14:58:25 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001366716.msg for ; Thu, 28 Apr 2005 15:54:01 +0100 Message-ID: <06d301c54c02$98ab5ce0$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Eric Anderson" References: <069901c54bfd$2967ba40$7f06000a@int.mediasurface.com> <4270F648.5030501@centtech.com> Date: Thu, 28 Apr 2005 15:57:28 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 15:54:01 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 15:54:02 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 14:58:26 -0000 ----- Original Message ----- From: "Eric Anderson" >> Where do I start looking? > First, understand that RAID 5 is dependant on fast hardware to performa > the XOR operations. A single disk without any RAID can easily > outperform a RAID array if the RAID array is on a 'slow' controller. > The Highpoint controllers are not exactly high-end fast RAID > controllers, so ~50MB/s isn't too bad for that I would say. Did you > happen to try a vinum RAID5? How about a stripe? The quoted benchmarks for this controller are ~500MB/s with 8 raptor disks. Its a hardware RAID 5 with 8 SATA channels on 64Bit PCI-X @ 133Mhz so under the current setup it should be achieving ~250MB/s Im only getting 50MB/s which is just silly. It was also my inderstanding that the XOR is only done on write not read. > You could make two RAID 5's with the controller, and then stripe those > in vinum. You could also do a combo of two stripes in the controller, > and a vinum mirror in FreeBSD - that would give you decent performance. I could but it seems there is something more problematic a foot here and would like to investigate the reason for the poor performance not workaround it. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 15:48:30 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A4AB16A4CE for ; Thu, 28 Apr 2005 15:48:30 +0000 (GMT) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 42D1043D1D for ; Thu, 28 Apr 2005 15:48:29 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j3SFmSNB034879; Thu, 28 Apr 2005 10:48:28 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42710594.2010308@centtech.com> Date: Thu, 28 Apr 2005 10:47:32 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <069901c54bfd$2967ba40$7f06000a@int.mediasurface.com> <4270F648.5030501@centtech.com> <06d301c54c02$98ab5ce0$7f06000a@int.mediasurface.com> In-Reply-To: <06d301c54c02$98ab5ce0$7f06000a@int.mediasurface.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 15:48:30 -0000 Steven Hartland wrote: > ----- Original Message ----- From: "Eric Anderson" > >>> Where do I start looking? > > >> First, understand that RAID 5 is dependant on fast hardware to >> performa the XOR operations. A single disk without any RAID can >> easily outperform a RAID array if the RAID array is on a 'slow' >> controller. The Highpoint controllers are not exactly high-end fast >> RAID controllers, so ~50MB/s isn't too bad for that I would say. Did >> you happen to try a vinum RAID5? How about a stripe? > > > The quoted benchmarks for this controller are ~500MB/s with 8 raptor > disks. Its a hardware RAID 5 with 8 SATA channels on 64Bit PCI-X > @ 133Mhz so under the current setup it should be achieving ~250MB/s > Im only getting 50MB/s which is just silly. > > It was also my inderstanding that the XOR is only done on write > not read. Correct - I misread the dd line. When you are doing the dd, what is your system busy doing? (top/ps info) >> You could make two RAID 5's with the controller, and then stripe those >> in vinum. You could also do a combo of two stripes in the controller, >> and a vinum mirror in FreeBSD - that would give you decent performance. > > > I could but it seems there is something more problematic a foot > here and would like to investigate the reason for the poor performance > not workaround it. What do you suspect? Some info from one of my arrays: # dd if=/dev/zero of=./testfile bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 3.581413 secs (182989226 bytes/sec) (unmount / mount filesystem) # dd if=./testfile of=/dev/null bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 6.747707 secs (97123362 bytes/sec) # dd if=/dev/da0s1d of=/dev/null bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 11.252226 secs (58242699 bytes/sec) Seems like the reads should be faster than the writes.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 15:59:08 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6F17816A4CE for ; Thu, 28 Apr 2005 15:59:08 +0000 (GMT) Received: from web41201.mail.yahoo.com (web41201.mail.yahoo.com [66.218.93.34]) by mx1.FreeBSD.org (Postfix) with SMTP id 3949F43D46 for ; Thu, 28 Apr 2005 15:59:08 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 41483 invoked by uid 60001); 28 Apr 2005 15:59:08 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=6dxqVCndG5fafemVL5TF/ZGaZKw3JIjLrMjVtUyEJgzo3XigWNzXX/jp77NuHfbOtWQvq1q+Z+bciVzobgEqLd8rhO/n/XfAnDYXhfiDeBYaHhc5h3W56LM1HXKZstB46167jFlMTr2J6rza+pxe5jVKkrL64dfkaRxeD9RAW9c= ; Message-ID: <20050428155908.41481.qmail@web41201.mail.yahoo.com> Received: from [83.129.189.236] by web41201.mail.yahoo.com via HTTP; Thu, 28 Apr 2005 08:59:07 PDT Date: Thu, 28 Apr 2005 08:59:07 -0700 (PDT) From: Arne "Wörner" To: Eric Anderson In-Reply-To: <4270F648.5030501@centtech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 15:59:08 -0000 --- Eric Anderson wrote: > First, understand that RAID 5 is dependant on fast hardware to > performa > the XOR operations. A single disk without any RAID can easily > outperform a RAID array if the RAID array is on a 'slow' > controller. > The Highpoint controllers are not exactly high-end fast RAID > controllers, so ~50MB/s isn't too bad for that I would say. > Did you happen to try a vinum RAID5? How about a stripe? > Furthermore RAID-5 needs to read the parity block, before it can update that block, so that there are 2 disc transactions more, which could explain the better performance of a single disk, too? Or does the striping effect (inherent to RAID-5) compensate those 2 extra transactions (at least in case of sequential writes)? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 16:04:32 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9B28C16A4CE for ; Thu, 28 Apr 2005 16:04:32 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0935C43D3F for ; Thu, 28 Apr 2005 16:04:32 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001366900.msg for ; Thu, 28 Apr 2005 16:59:58 +0100 Message-ID: <070201c54c0b$cd889a00$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: =?iso-8859-1?Q?Arne_W=F6rner?= , "Eric Anderson" References: <20050428155908.41481.qmail@web41201.mail.yahoo.com> Date: Thu, 28 Apr 2005 17:03:22 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 16:59:58 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 16:59:58 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 16:04:32 -0000 Only on write this is a read test. ----- Original Message ----- From: "Arne Wörner" > Furthermore RAID-5 needs to read the parity block, before it can > update that block, so that there are 2 disc transactions more, > which could explain the better performance of a single disk, too? > > Or does the striping effect (inherent to RAID-5) compensate those > 2 extra transactions (at least in case of sequential writes)? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 16:08:48 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7402316A4CE for ; Thu, 28 Apr 2005 16:08:48 +0000 (GMT) Received: from web41203.mail.yahoo.com (web41203.mail.yahoo.com [66.218.93.36]) by mx1.FreeBSD.org (Postfix) with SMTP id 4618743D1D for ; Thu, 28 Apr 2005 16:08:48 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 13544 invoked by uid 60001); 28 Apr 2005 16:08:47 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=MLhAlAMqwtQUR/JV6bWMizJxmnWpQVJKn7GJ7okFOALAyW2xpLONrFNWsqTL/qrDG40C3pG0Au0hbBS1Y2h0arQ2A5Sh+DgUva4Dq5NKuq9b41qSQM5VscBcT+HFRD8NTPoxqalZYmOz3OrCSzoDfczSplFxgoAak0Xx1oryaPo= ; Message-ID: <20050428160847.13542.qmail@web41203.mail.yahoo.com> Received: from [83.129.189.236] by web41203.mail.yahoo.com via HTTP; Thu, 28 Apr 2005 09:08:47 PDT Date: Thu, 28 Apr 2005 09:08:47 -0700 (PDT) From: Arne "Wörner" To: Steven Hartland , Eric Anderson In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 16:08:48 -0000 --- Steven Hartland wrote: > From: "Eric Anderson" > >> Where do I start looking? > > First, understand that RAID 5 is dependant on fast hardware to > > performa > > the XOR operations. A single disk without any RAID can easily > > outperform a RAID array if the RAID array is on a 'slow' > > controller. > > The Highpoint controllers are not exactly high-end fast RAID > > controllers, so ~50MB/s isn't too bad for that I would say. > > Did you > > happen to try a vinum RAID5? How about a stripe? > > The quoted benchmarks for this controller are ~500MB/s with 8 > raptor disks. Its a hardware RAID 5 with 8 SATA channels on > 64Bit PCI-X > @ 133Mhz so under the current setup it should be achieving > ~250MB/s > Im only getting 50MB/s which is just silly. > There was a discussion some months ago. It seemed like ata disc i/o throughput is in R5 abou 50% slower than in R4. I was able to see that here in my setting, too. But I have to admit, that my hard discs are probably confused by my experiments with the write-cache-disable sysctl... > It was also my inderstanding that the XOR is only done on write > not read. > Yes! The data can be constructed without the parity block, if all other discs are available. But it might be faster, to use the parity block in some rare cases (e. g.: the disc that holds parity data is idle but the others are not AND all corresponding blocks but one are already in cache memory :-) ). > > You could make two RAID 5's with the controller, and then > > stripe those > > in vinum. You could also do a combo of two stripes in the > > controller, > > and a vinum mirror in FreeBSD - that would give you decent > > performance. > > I could but it seems there is something more problematic a foot > here and would like to investigate the reason for the poor > performance not workaround it. > Did you try RedHat Linux or FreeBSD R4? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 16:08:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A3D9D16A4F8 for ; Thu, 28 Apr 2005 16:08:52 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 03CFF43D41 for ; Thu, 28 Apr 2005 16:08:52 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001366916.msg for ; Thu, 28 Apr 2005 17:04:44 +0100 Message-ID: <070d01c54c0c$79550260$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Eric Anderson" References: <069901c54bfd$2967ba40$7f06000a@int.mediasurface.com> <4270F648.5030501@centtech.com> <06d301c54c02$98ab5ce0$7f06000a@int.mediasurface.com> <42710594.2010308@centtech.com> Date: Thu, 28 Apr 2005 17:08:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 17:04:44 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 17:04:46 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 16:08:52 -0000 ----- Original Message ----- From: "Eric Anderson" > Correct - I misread the dd line. When you are doing the dd, what is > your system busy doing? (top/ps info) The machine is idle only me doing the test via an ssh session. > What do you suspect? I really dont know what it could be driver issue? iirc scott wrote / converted this one from source direct form highpoint so maybe he could have some useful imput here. > Some info from one of my arrays: > > # dd if=/dev/zero of=./testfile bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 3.581413 secs (182989226 bytes/sec) > > (unmount / mount filesystem) > # dd if=./testfile of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 6.747707 secs (97123362 bytes/sec) > > # dd if=/dev/da0s1d of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 11.252226 secs (58242699 bytes/sec) > > Seems like the reads should be faster than the writes.. Hmm just tried the write and I get: dd if=/dev/zero of=./testfile bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 5.237026 secs (125139727 bytes/sec) Which is 2.5 times quicker than read which is really just not right. Issue with /dev/null? Steve / K ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 16:22:02 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1B53116A4CE for ; Thu, 28 Apr 2005 16:22:02 +0000 (GMT) Received: from web41214.mail.yahoo.com (web41214.mail.yahoo.com [66.218.93.47]) by mx1.FreeBSD.org (Postfix) with SMTP id B254943D31 for ; Thu, 28 Apr 2005 16:22:01 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 39271 invoked by uid 60001); 28 Apr 2005 16:22:01 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=ZhUzdL30egARs8w37gecUOqxqRO8GbEYNnaH0h8iESbUIlKMAL3Uoc2LpDqNRQ5EPFw2VpJqJPz2/xtLu501DIwQSgvxmqwtUSW+ayW99S1EFkgkJpcYTKZywKY+vr/Kjtdasx74MRQhK7yz4x7wos8lGozccoZgJ1IAFPtHztI= ; Message-ID: <20050428162201.39269.qmail@web41214.mail.yahoo.com> Received: from [83.129.189.236] by web41214.mail.yahoo.com via HTTP; Thu, 28 Apr 2005 09:22:01 PDT Date: Thu, 28 Apr 2005 09:22:01 -0700 (PDT) From: Arne "Wörner" To: Steven Hartland , Eric Anderson In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 16:22:02 -0000 --- Steven Hartland wrote: > Only on write this is a read test. > Oh. Ok! :-) But there is a striping-like effect (especially when u use n=8 discs in one RAID-5 and when u do sequential read), so that the performance could be easily (n-1)*40MB/sec (which would be in your case 350MB/sec; my single ATA disc reads 40 or about 40 MB/sec). So I would say, that something in the ATA driver is sub-optimal and worse than before. But I do not know what... There are so many lines... -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 17:03:27 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C7AFB16A4CE for ; Thu, 28 Apr 2005 17:03:27 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB0BA43D54 for ; Thu, 28 Apr 2005 17:03:26 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j3SH3P0F066218; Thu, 28 Apr 2005 12:03:26 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42711726.2000607@centtech.com> Date: Thu, 28 Apr 2005 12:02:30 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= References: <20050428162201.39269.qmail@web41214.mail.yahoo.com> In-Reply-To: <20050428162201.39269.qmail@web41214.mail.yahoo.com> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: ClamAV 0.82/858/Thu Apr 28 09:02:38 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org cc: Steven Hartland Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 17:03:27 -0000 Arne W=F6rner wrote: > --- Steven Hartland wrote: >=20 >>Only on write this is a read test. >> >=20 > Oh. Ok! :-) >=20 > But there is a striping-like effect (especially when u use n=3D8 > discs in one RAID-5 and when u do sequential read), so that the > performance could be easily (n-1)*40MB/sec (which would be in your > case 350MB/sec; my single ATA disc reads 40 or about 40 MB/sec). >=20 > So I would say, that something in the ATA driver is sub-optimal > and worse than before. But I do not know what... There are so many > lines... I don't think it is ATA related - my tests were through a Fiber Channel=20 card. My reads were 1/2 of my writes.. Eric --=20 ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 17:11:44 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA9E716A4CE for ; Thu, 28 Apr 2005 17:11:44 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 142AF43D1D for ; Thu, 28 Apr 2005 17:11:44 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001367116.msg for ; Thu, 28 Apr 2005 18:07:11 +0100 Message-ID: <077d01c54c15$32900290$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: =?iso-8859-1?Q?Arne_W=F6rner?= , "Eric Anderson" References: <20050428160847.13542.qmail@web41203.mail.yahoo.com> Date: Thu, 28 Apr 2005 18:10:37 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 18:07:11 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 18:07:11 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 17:11:44 -0000 ----- Original Message ----- From: "Arne Wörner" > Did you try RedHat Linux or FreeBSD R4? Haven't tried R4 or Linux yet. Just finished restoring 700GB onto the machine and would rather not have to do that again :) Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Apr 28 23:00:42 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E557916A4CE for ; Thu, 28 Apr 2005 23:00:41 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 007D843D54 for ; Thu, 28 Apr 2005 23:00:41 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001367806.msg for ; Thu, 28 Apr 2005 23:57:09 +0100 Message-ID: <006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" ,, "Scott Long" References: <20050428162201.39269.qmail@web41214.mail.yahoo.com> <42711726.2000607@centtech.com> Date: Fri, 29 Apr 2005 00:00:28 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-15"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Thu, 28 Apr 2005 23:57:09 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Thu, 28 Apr 2005 23:57:11 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2005 23:00:42 -0000 Scott I've sent this to you as from reading around you did the original driver conversion and as such may have an idea on the areas I could look at hope you dont mind. Ok some real strange going on write performance is ~ 140MB/s: gstat: dT: 0.505 flag_I 500000us sizeof 240 i -1 L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 1100 4 63 13.2 1096 140313 1.2 57.8| da0 0 1100 4 63 13.3 1096 140313 1.3 59.3| da0s1 where as read is ~42MB/s gstat: dT: 0.505 flag_I 500000us sizeof 240 i -1 L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 1 335 335 42836 2.8 0 0 0.0 93.3| da0 1 335 335 42836 2.8 0 0 0.0 93.6| da0s1 Some updated figures: 5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) Write: dd if=/dev/zero of=./test.0 bs=64k count=50000 50000+0 records in 50000+0 records out 3276800000 bytes transferred in 25.088439 secs (130609959 bytes/sec) Read: dd if=./test.0 of=/dev/null bs=64k count=50000 50000+0 records in 50000+0 records out 3276800000 bytes transferred in 69.765394 secs (46968845 bytes/sec) vmstat while doing the above dd: Read: vmstat -w 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 0 1 0 56560 1840332 6 2 0 0 872 539 0 0 1625 1859 3468 1 3 96 0 1 0 56560 1797548 3 0 0 0 3 0 331 0 1949 1449 7071 0 7 93 0 1 0 56560 1751308 0 0 0 0 0 0 359 0 2071 1581 7600 0 7 93 0 1 0 56560 1707516 0 0 0 0 0 0 338 0 1974 1488 7219 0 8 92 0 1 0 56560 1663084 0 0 0 0 0 0 343 0 2003 1509 7300 0 10 90 .... Write: vmstat -w 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 pa0 in sy cs us sy id 0 0 1 56560 1912484 6 1 0 0 804 495 1246 0 1627 1864 3454 1 3 96 0 0 1 56560 1779912 3 0 0 0 3 0 1015 0 3606 4272 18207 0 94 6 0 0 1 56560 1634632 0 0 0 0 0 0 1123 0 3939 4657 17973 0 51 49 0 0 1 56560 1507016 0 0 0 0 0 0 996 0 3522 4104 15487 0 46 54 0 0 1 56560 1367688 0 0 0 0 0 0 1075 0 3748 4469 17131 0 45 55 ... iostat while doing the above dd: Read: tty da0 pass0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 61 127.66 328 40.94 0.00 0 0.00 0 0 7 1 92 0 61 127.39 365 45.43 0.00 0 0.00 0 0 6 1 92 0 61 127.13 383 47.56 0.00 0 0.00 0 0 8 1 91 ... Write: tty da0 pass0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 62 126.26 1031 127.18 0.00 0 0.00 0 0 34 13 53 0 63 127.67 1004 125.19 0.00 0 0.00 0 0 35 12 53 0 63 127.57 1028 128.05 0.00 0 0.00 0 0 38 12 51 0 63 127.57 1025 127.63 0.00 0 0.00 0 0 34 11 55 System Summary: Dual AMD 244, 2GB ram, 5 * Seagate SATA 400GB on a Highpoint 1820a 64Bit PCI-X @ 133Mhz RAID 5 array. Relavent dmesg: FreeBSD 5.4-STABLE #0: Wed Apr 27 08:10:14 UTC 2005 root@test:/usr/src/sys/i386/compile/SMP_200HZ Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 244 (1794.41-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10 Features=0x78bfbff AMD Features=0xe0500000 real memory = 2146893824 (2047 MB) avail memory = 2099625984 (2002 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 hptmv0: mem 0xf8200000-0xf827ffff irq 25 at device 1.0 on pci9 RocketRAID 182x SATA Controller driver Version 1.1 RR182x [0,0]: channel started successfully RR182x [0,1]: channel started successfully RR182x [0,2]: channel started successfully RR182x [0,3]: channel started successfully RR182x [0,4]: channel started successfully RR182x: RAID5 write-back enabled Timecounters tick every 5.000 msec da0 at hptmv0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 1526216MB (3125690368 512 byte sectors: 255H 63S/T 194565C) Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 00:34:53 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AEED016A4CE; Fri, 29 Apr 2005 00:34:53 +0000 (GMT) Received: from farside.isc.org (farside.isc.org [204.152.187.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8AB3543D60; Fri, 29 Apr 2005 00:34:53 +0000 (GMT) (envelope-from Peter_Losher@isc.org) Received: from [IPv6:2001:4f8:3:bb::37] (tardis.isc.org [IPv6:2001:4f8:3:bb::37]) by farside.isc.org (Postfix) with ESMTP id 67C74677FA; Fri, 29 Apr 2005 00:34:53 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Message-ID: <4271812F.4060808@isc.org> Date: Thu, 28 Apr 2005 17:34:55 -0700 From: Peter Losher User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050325) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> In-Reply-To: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> X-Enigmail-Version: 0.91.0.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigFA6731D71D705328E94F2D22" cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 00:34:53 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigFA6731D71D705328E94F2D22 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Steven Hartland wrote: > 5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 13.348032 secs (49097875 bytes/sec) Have you tried the driver supplied by Highpoint as a pre-compiled kernel module? I have a 1820A with 4x400GB Hitachi SATA disks (RAID5 + 1 Hot Spare) in a system running 5.3-RC3/amd64, where I have to use the kernel module, and I get almost double the speed that you have: -=- % dd if=/dev/da0 of=/dev/null bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 7.775190 secs (84288614 bytes/sec) -=- Best Wishes - Peter -- Peter_Losher@isc.org | ISC | OpenPGP 0xE8048D08 | "The bits must flow" --------------enigFA6731D71D705328E94F2D22 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iD8DBQFCcYEvPtVx9OgEjQgRAlF+AKCQT6REpmtZFXzBE2jJnaRLJcyxHwCgsmVS NXe3yLTwG2fg2KAwK7+41HE= =MoJJ -----END PGP SIGNATURE----- --------------enigFA6731D71D705328E94F2D22-- From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 03:45:30 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E148A16A4CE; Fri, 29 Apr 2005 03:45:30 +0000 (GMT) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id EF28743D54; Fri, 29 Apr 2005 03:45:29 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.51.77] ([192.168.51.77]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j3T3jSWZ041344; Thu, 28 Apr 2005 22:45:28 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <4271ADA0.2070000@centtech.com> Date: Thu, 28 Apr 2005 22:44:32 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Losher References: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> <4271812F.4060808@isc.org> In-Reply-To: <4271812F.4060808@isc.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: Steven Hartland Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 03:45:31 -0000 Peter Losher wrote: > Steven Hartland wrote: > > >>5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) >>dd if=/dev/da0 of=/dev/null bs=64k count=10000 >>10000+0 records in >>10000+0 records out >>655360000 bytes transferred in 13.348032 secs (49097875 bytes/sec) > > > Have you tried the driver supplied by Highpoint as a pre-compiled kernel > module? I have a 1820A with 4x400GB Hitachi SATA disks (RAID5 + 1 Hot > Spare) in a system running 5.3-RC3/amd64, where I have to use the kernel > module, and I get almost double the speed that you have: > > -=- > % dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 7.775190 secs (84288614 bytes/sec) I'm using fiber channel SATA, and I get 2x write as I do read, which doesn't make sense to me. What kind of write speeds do you get? My tiny brain tells me that reads should be faster than writes with a RAID5. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 04:09:04 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8F6FE16A4CE; Fri, 29 Apr 2005 04:09:04 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id CEB3143D54; Fri, 29 Apr 2005 04:09:03 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (8.13.1/8.11.4) with ESMTP id j3T48sdE097207; Fri, 29 Apr 2005 07:08:54 +0300 (EEST) (envelope-from pete@he.iki.fi) Message-ID: <4271B378.5050100@he.iki.fi> Date: Fri, 29 Apr 2005 07:09:28 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Eric Anderson References: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> <4271812F.4060808@isc.org> <4271ADA0.2070000@centtech.com> In-Reply-To: <4271ADA0.2070000@centtech.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: Steven Hartland cc: Peter Losher Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 04:09:04 -0000 Eric Anderson wrote: > > I'm using fiber channel SATA, and I get 2x write as I do read, which > doesn't make sense to me. What kind of write speeds do you get? My > tiny brain tells me that reads should be faster than writes with a RAID5. > I'm seeing similar sequential performance on RELENG_5_3 and RELENG_5_4 on dual-Xeons using 3ware controllers so it does not seem to be a driver issue but somewhere else in the architechture. Depending on the array configuration, 40-60MB/s reads and 100-160MB/s writes. The write performance is as expected but the read performance should be above the write performance. Pete From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 08:12:27 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE8F016A4CE for ; Fri, 29 Apr 2005 08:12:27 +0000 (GMT) Received: from web41207.mail.yahoo.com (web41207.mail.yahoo.com [66.218.93.40]) by mx1.FreeBSD.org (Postfix) with SMTP id BEEF343D31 for ; Fri, 29 Apr 2005 08:12:27 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 11497 invoked by uid 60001); 29 Apr 2005 08:12:27 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=IYV4uRvUNpKL/Q0E39sKh1xv15FVTY4yjbf76lK0UbpNDtFznxQUY47lOvQcu9oLLSJ5opV/8Vhyhn5ujJy8RWyOyN6XxuWv3Mlyh38P7kuHyme8dDo4uKnuxS1ba0/HsxHIgpkfc+KLOX/fg365l0WkE7hr8dhzQLIP7AxZrTA= ; Message-ID: <20050429081227.11495.qmail@web41207.mail.yahoo.com> Received: from [213.54.151.37] by web41207.mail.yahoo.com via HTTP; Fri, 29 Apr 2005 01:12:27 PDT Date: Fri, 29 Apr 2005 01:12:27 -0700 (PDT) From: Arne "Wörner" To: Petri Helenius , freebsd-performance@freebsd.org In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 08:12:28 -0000 --- Petri Helenius wrote: > Eric Anderson wrote: > I'm seeing similar sequential performance on RELENG_5_3 and > RELENG_5_4 > on dual-Xeons using 3ware controllers so it does not seem to be > a driver issue [...] > Why? I can remember, that some people said some months before, that FreeBSD's R4 was much faster (maybe something in the kernel changed due to SMPng, so that the disc subsystem behaves funny?). -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 08:27:21 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ED63C16A4CE for ; Fri, 29 Apr 2005 08:27:21 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18D1E43D54 for ; Fri, 29 Apr 2005 08:27:19 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j3T8V749028105; Fri, 29 Apr 2005 02:31:08 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4271EFCA.1060605@samsco.org> Date: Fri, 29 Apr 2005 02:26:50 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <20050428162201.39269.qmail@web41214.mail.yahoo.com> <42711726.2000607@centtech.com> <006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> In-Reply-To: <006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=5C?= =?ISO-8859-1?Q?W=F6rner=5C=22=22?= Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 08:27:22 -0000 Steven Hartland wrote: > Scott I've sent this to you as from reading around you did the > original driver conversion and as such may have an idea > on the areas I could look at hope you dont mind. > > Ok some real strange going on write performance is ~ 140MB/s: > gstat: > dT: 0.505 flag_I 500000us sizeof 240 i -1 > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 1100 4 63 13.2 1096 140313 1.2 57.8| da0 > 0 1100 4 63 13.3 1096 140313 1.3 59.3| da0s1 > > where as read is ~42MB/s > gstat: > dT: 0.505 flag_I 500000us sizeof 240 i -1 > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 1 335 335 42836 2.8 0 0 0.0 93.3| da0 > 1 335 335 42836 2.8 0 0 0.0 93.6| da0s1 > First of all, you're only sending roughly 3GB of data through. Since you have 2GB of RAM, you're likely getting a lot of write caching from the OS. If you want more representative numbers, either test with a much larger data set or with a much smaller RAM size. Second, since you're going through the filesystem, there is a very good chance that the filesystem blocks are not aligning well with the array blocks. This hurts quite a bit on any controller, and I can imagine it being extremely bad on a controller like this one. Try doing your DD test straight to the device node. In my local testing I was able to get about 400MB/sec across 6 disks in RAID-0. RAID-5 read should get almost as good in a similar configuration (unless the RAID stack is checking parity on read, a question that I cannot answer). I would expect RAID-5 write to be significantly lower due to the extra cpu and memory bus overhead of doing the parity calculations. This will of course depend also on the speed of your drives and the speed of your PCI and memory bus. The DD commands that I usually use: dd if=/dev/zero of=/dev/da0 bs=1m dd if=/dev/da0 of=/dev/null bs=1m From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 09:40:22 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AFE7E16A4CE for ; Fri, 29 Apr 2005 09:40:22 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id D23F943D1D for ; Fri, 29 Apr 2005 09:40:21 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [195.163.185.142] (i2-142.rommon.fi [195.163.185.142]) by silver.he.iki.fi (Postfix) with ESMTP id 2F1E0BC40; Fri, 29 Apr 2005 12:40:19 +0300 (EEST) Message-ID: <42720125.3090201@he.iki.fi> Date: Fri, 29 Apr 2005 12:40:53 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Arne \"WXrner\"" References: <20050429081227.11495.qmail@web41207.mail.yahoo.com> In-Reply-To: <20050429081227.11495.qmail@web41207.mail.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 09:40:22 -0000 Arne WXrner wrote: >--- Petri Helenius wrote: > > >>Eric Anderson wrote: >>I'm seeing similar sequential performance on RELENG_5_3 and >>RELENG_5_4 >>on dual-Xeons using 3ware controllers so it does not seem to be >>a driver issue [...] >> >> >> >Why? > >I can remember, that some people said some months before, that >FreeBSD's R4 was much faster (maybe something in the kernel >changed due to SMPng, so that the disc subsystem behaves funny?). > > I mean the issue does not seem to be specific to a controller, but generic in the way how kernel schedules requests. Pete From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 10:18:27 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 000AC16A4CE; Fri, 29 Apr 2005 10:18:26 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0FA6743D2D; Fri, 29 Apr 2005 10:18:26 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001368651.msg; Fri, 29 Apr 2005 11:14:53 +0100 Message-ID: <08a501c54ca4$c3e80f00$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Peter Losher" References: <06ba01c54bff$8c5aed00$7f06000a@int.mediasurface.com> <4271812F.4060808@isc.org> Date: Fri, 29 Apr 2005 11:18:19 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Fri, 29 Apr 2005 11:14:53 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Fri, 29 Apr 2005 11:14:53 +0100 cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 10:18:27 -0000 There is no precompiled version for 5.3 but looking at the openbuild version its the same driver as the built in. 80MB/s is still terrible should be looking closer to 200MB/s. Steven Hartland wrote: > 5.4-STABLE Highpoint 1820a RAID 5 ( 5 disk ) > dd if=/dev/da0 of=/dev/null bs=64k count=10000 > 10000+0 records in > 10000+0 records out > 655360000 bytes transferred in 13.348032 secs (49097875 bytes/sec) Have you tried the driver supplied by Highpoint as a pre-compiled kernel module? I have a 1820A with 4x400GB Hitachi SATA disks (RAID5 + 1 Hot Spare) in a system running 5.3-RC3/amd64, where I have to use the kernel module, and I get almost double the speed that you have: -=- % dd if=/dev/da0 of=/dev/null bs=64k count=10000 10000+0 records in 10000+0 records out 655360000 bytes transferred in 7.775190 secs (84288614 bytes/sec) ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 10:58:33 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 783D716A4CE for ; Fri, 29 Apr 2005 10:58:33 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8FA2543D5D for ; Fri, 29 Apr 2005 10:58:32 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001368772.msg for ; Fri, 29 Apr 2005 11:53:43 +0100 Message-ID: <093701c54caa$303aac30$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "Scott Long" References: <20050428162201.39269.qmail@web41214.mail.yahoo.com><42711726.2000607@centtech.com><006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> <4271EFCA.1060605@samsco.org> Date: Fri, 29 Apr 2005 11:57:08 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Fri, 29 Apr 2005 11:53:43 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 29 Apr 2005 11:54:17 +0100 cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: Petri Helenius cc: =?iso-8859-1?Q?=22Arne_\\W=F6rner\=22=22?= Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 10:58:33 -0000 ----- Original Message ----- From: "Scott Long" >> Ok some real strange going on write performance is ~ 140MB/s: >> gstat: >> dT: 0.505 flag_I 500000us sizeof 240 i -1 >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name >> 0 1100 4 63 13.2 1096 140313 1.2 57.8| da0 >> 0 1100 4 63 13.3 1096 140313 1.3 59.3| da0s1 >> >> where as read is ~42MB/s >> gstat: >> dT: 0.505 flag_I 500000us sizeof 240 i -1 >> L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name >> 1 335 335 42836 2.8 0 0 0.0 93.3| da0 >> 1 335 335 42836 2.8 0 0 0.0 93.6| da0s1 > > First of all, you're only sending roughly 3GB of data through. Since > you have 2GB of RAM, you're likely getting a lot of write caching > from the OS. If you want more representative numbers, either test > with a much larger data set or with a much smaller RAM size. Second, > since you're going through the filesystem, there is a very good chance > that the filesystem blocks are not aligning well with the array > blocks. This hurts quite a bit on any controller, and I can imagine > it being extremely bad on a controller like this one. Try doing your > DD test straight to the device node. In my local testing I was able to > get about 400MB/sec across 6 disks in RAID-0. RAID-5 read should > get almost as good in a similar configuration (unless the RAID stack > is checking parity on read, a question that I cannot answer). I would > expect RAID-5 write to be significantly lower due to the extra cpu and > memory bus overhead of doing the parity calculations. This will of > course depend also on the speed of your drives and the speed of your PCI > and memory bus. > > The DD commands that I usually use: > > dd if=/dev/zero of=/dev/da0 bs=1m > dd if=/dev/da0 of=/dev/null bs=1m Just retried with a ~10Gb data set: Write to FS: dd if=/dev/zero of=.testfile bs=1m count=10000 10000+0 records in 10000+0 records out 10485760000 bytes transferred in 92.517222 secs (113338466 bytes/sec) Read from FS: dd if=.testfile of=/dev/null bs=1m count=10000 10000+0 records in 10000+0 records out 10485760000 bytes transferred in 225.723348 secs (46454034 bytes/sec) Read from device: dd if=/dev/da0 of=/dev/null bs=1m count=10000 10000+0 records in 10000+0 records out 10485760000 bytes transferred in 200.723666 secs (52239779 bytes/sec) N.B. didn't do the write to the device direct as the array is the system drive so I don't think it would take kindly to that :) So it doesn't seem like caching is an issue and as others are seeing similar performance issues on other RAID controllers is could well not be a driver issue but I'm not ruling that out as yet as it could be the same problem in each respective driver. So far Highpoint 1820a and 3ware ?? ( Pete can u fill in the blank here ) are exhibiting the same issue. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 13:14:24 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 63C0816A4CE for ; Fri, 29 Apr 2005 13:14:24 +0000 (GMT) Received: from ms1.as.pvp.se (dns.pvp.se [213.64.187.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F9B443D5D for ; Fri, 29 Apr 2005 13:14:23 +0000 (GMT) (envelope-from kama@pvp.se) Received: by ms1.as.pvp.se (Postfix, from userid 1001) id 5DD54A7; Fri, 29 Apr 2005 15:14:21 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by ms1.as.pvp.se (Postfix) with ESMTP id 59BBDA6; Fri, 29 Apr 2005 15:14:21 +0200 (CEST) Date: Fri, 29 Apr 2005 15:14:21 +0200 (CEST) From: kama X-X-Sender: kama@ns1.as.pvp.se To: Steven Hartland In-Reply-To: <093701c54caa$303aac30$7f06000a@int.mediasurface.com> Message-ID: <20050429150534.L22614@ns1.as.pvp.se> References: <20050428162201.39269.qmail@web41214.mail.yahoo.com><42711726.2000607@centtech.com><006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> <093701c54caa$303aac30$7f06000a@int.mediasurface.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Scott Long cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: =?iso-8859-1?Q?=22Arne_W=F6rner=22=22?= cc: Petri Helenius Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 13:14:24 -0000 On Fri, 29 Apr 2005, Steven Hartland wrote: > Just retried with a ~10Gb data set: > Write to FS: > dd if=/dev/zero of=.testfile bs=1m count=10000 > 10000+0 records in > 10000+0 records out > 10485760000 bytes transferred in 92.517222 secs (113338466 bytes/sec) > > Read from FS: > dd if=.testfile of=/dev/null bs=1m count=10000 > 10000+0 records in > 10000+0 records out > 10485760000 bytes transferred in 225.723348 secs (46454034 bytes/sec) > > Read from device: > dd if=/dev/da0 of=/dev/null bs=1m count=10000 > 10000+0 records in > 10000+0 records out > 10485760000 bytes transferred in 200.723666 secs (52239779 bytes/sec) > > N.B. didn't do the write to the device direct as the array is > the system drive so I don't think it would take kindly to that :) > > So it doesn't seem like caching is an issue and as others are seeing > similar performance issues on other RAID controllers is could well > not be a driver issue but I'm not ruling that out as yet as it could be > the same problem in each respective driver. > So far Highpoint 1820a and 3ware ?? ( Pete can u fill in the blank here ) > are exhibiting the same issue. > I have just tested on my system between 4.11 and 5.4. Both uses the same configuration on the write disk. its a RAID1+0 on Compaq SA6404 (ciss) running UFS1. The write to disk is similar on 4.11 and 5.4, but the read is 100% slower on 5.4 than 4.11. the R:W condition on 4.11 is 1:1 and on 5.4 2:1. I have tested this with dd and my own written disktest program. The results on both program is similar. so it isn't dev-null issues. I have tested with both 1GB files and 10GB files. 1GB files where really fast on reading, but i consider them false due to the RAID cache. Something is really wrong here... /Bjorn From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 13:22:45 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1F82516A4CE for ; Fri, 29 Apr 2005 13:22:45 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C70D43D4C for ; Fri, 29 Apr 2005 13:22:44 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from stevenp4 ([193.123.241.40]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001369247.msg for ; Fri, 29 Apr 2005 14:18:37 +0100 Message-ID: <09e001c54cbe$6eaa7810$7f06000a@int.mediasurface.com> From: "Steven Hartland" To: "kama" References: <20050428162201.39269.qmail@web41214.mail.yahoo.com><42711726.2000607@centtech.com><006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> <4271EFCA.1060605@samsco.org> <093701c54caa$303aac30$7f06000a@int.mediasurface.com> <20050429150534.L22614@ns1.as.pvp.se> Date: Fri, 29 Apr 2005 14:22:02 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Fri, 29 Apr 2005 14:18:37 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 193.123.241.40 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Fri, 29 Apr 2005 14:18:38 +0100 cc: Scott Long cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: =?iso-8859-1?Q?=22Arne_W=F6rner=22=22?= cc: Petri Helenius Subject: Re: Very low disk read performance on 5.x ( was Very low disk performance Highpoint 1820a ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 13:22:45 -0000 Ok thanks for that kama good to have some comparison with 4.x I've changed the subject as this seems definitely like a more generic issue something that needs to be fixed before 5.4 release? ----- Original Message ----- From: "kama" > > I have just tested on my system between 4.11 and 5.4. Both uses the same > configuration on the write disk. its a RAID1+0 on Compaq SA6404 (ciss) > running UFS1. The write to disk is similar on 4.11 and 5.4, but the read > is 100% slower on 5.4 than 4.11. the R:W condition on 4.11 is 1:1 and on > 5.4 2:1. > > I have tested this with dd and my own written disktest program. The > results on both program is similar. so it isn't dev-null issues. > > I have tested with both 1GB files and 10GB files. 1GB files where really > fast on reading, but i consider them false due to the RAID cache. > > Something is really wrong here... ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 18:09:56 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE16116A4CE for ; Fri, 29 Apr 2005 18:09:56 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB4DD43D2F for ; Fri, 29 Apr 2005 18:09:55 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j3TI9s5C078388; Fri, 29 Apr 2005 13:09:54 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42727839.4030703@centtech.com> Date: Fri, 29 Apr 2005 13:08:57 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: killing@multiplay.co.uk References: <20050428162201.39269.qmail@web41214.mail.yahoo.com><42711726.2000607@centtech.com><006201c54c46$14c0d290$b3db87d4@multiplay.co.uk> <4271EFCA.1060605@samsco.org> <093701c54caa$303aac30$7f06000a@int.mediasurface.com> In-Reply-To: <093701c54caa$303aac30$7f06000a@int.mediasurface.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/860/Fri Apr 29 08:45:14 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance Highpoint 1820a X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 18:09:56 -0000 Steven Hartland wrote: > ----- Original Message ----- From: "Scott Long" > >>> Ok some real strange going on write performance is ~ 140MB/s: [..snip..] >>> where as read is ~42MB/s [..snip..] > So it doesn't seem like caching is an issue and as others are seeing > similar performance issues on other RAID controllers is could well > not be a driver issue but I'm not ruling that out as yet as it could be > the same problem in each respective driver. > So far Highpoint 1820a and 3ware ?? ( Pete can u fill in the blank here ) > are exhibiting the same issue. As I posted earlier also, I'm seeing the same behavior to a Fiber Channel RAID array (15 400GB SATA150 disks in RAID5 config). I'm using a QLogic HBA connected directly to the SATA array. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Fri Apr 29 22:10:03 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0D9AE16A4CE for ; Fri, 29 Apr 2005 22:10:03 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id C09CC43D54 for ; Fri, 29 Apr 2005 22:10:02 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 3219846B87 for ; Fri, 29 Apr 2005 18:10:02 -0400 (EDT) Date: Fri, 29 Apr 2005 23:12:46 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: performance@FreeBSD.org In-Reply-To: <20050425181101.Y74930@fledge.watson.org> Message-ID: <20050429231211.J31768@fledge.watson.org> References: <20050417134448.L85588@fledge.watson.org> <20050425114546.O74930@fledge.watson.org> <20050425181101.Y74930@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: Re: Memory allocation performance/statistics patches X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 22:10:03 -0000 On Mon, 25 Apr 2005, Robert Watson wrote: > On Mon, 25 Apr 2005, Robert Watson wrote: > >> I now have updated versions of these patches, which correct some >> inconsistencies in approach (universal use of curcpu now, for example), >> remove some debugging code, etc. I've received relatively little >> performance feedback on them, and would appreciate it if I could get some. >> :-) Especially as to whether these impact disk I/O related workloads, >> useful macrobenchmarks, etc. The latest patch is at: >> >> >> http://www.watson.org/~robert/freebsd/netperf/20050425-uma-mbuf-malloc-critical.diff > > FYI: For those set up to track perforce, you can find the contents of > this patch in: > > //depot/user/rwatson/percpu/... > > In addition, that branch also contains diagnostic micro-benchmarks in > the kernel to measure the cost of various synchronization operations, > memory allocation operations, etc, which can be queried using "sysctl > test". FYI: I've now committed the UMA changes from this patch. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sat Apr 30 21:12:31 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 37A3616A4CE for ; Sat, 30 Apr 2005 21:12:31 +0000 (GMT) Received: from web41203.mail.yahoo.com (web41203.mail.yahoo.com [66.218.93.36]) by mx1.FreeBSD.org (Postfix) with SMTP id F0E3643D2F for ; Sat, 30 Apr 2005 21:12:30 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 20780 invoked by uid 60001); 30 Apr 2005 21:12:30 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=179fATlVDQebc75dMDlJ0Bu1tVpNWt7SjJFCb+7NKxxJCkv3Y7OVtmfV7gz3r23M/zcvvRkcVyTSGodtFDnKzw8oIKaB4kc4LmfnoUbbaL1OWtrn9uLiLCzzIx/XfQ79YZ5cui8QIRtYIwpbzNrwc6HSTBa+6FdbioPUrHkBi6I= ; Message-ID: <20050430211230.20778.qmail@web41203.mail.yahoo.com> Received: from [213.54.153.164] by web41203.mail.yahoo.com via HTTP; Sat, 30 Apr 2005 14:12:30 PDT Date: Sat, 30 Apr 2005 14:12:30 -0700 (PDT) From: Arne "Wörner" To: Petri Helenius , freebsd-performance@freebsd.org In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Apr 2005 21:12:31 -0000 --- Arne Wörner wrote: > --- Petri Helenius wrote: > > Eric Anderson wrote: > > I'm seeing similar sequential performance on RELENG_5_3 > > and RELENG_5_4 on dual-Xeons using 3ware controllers so > > it does not seem to be a driver issue [...] > > > Why? > I found something out: 1. read and write speeds (same partition; hw.ata.wc=1; dd bs=128k count=2000; 512MB RAM): R4.11 R5.3 R 27.5MB/s 27.5MB/s W 27MB/s 5.15MB/s 2. The man page geom(4) does not exist in R4.11 3. The man page geom(4) of R5.3 says "The GEOM framework provides an infrastructure in which "classes" can per- form transformations on disk I/O requests on their path from the upper kernel to the device drivers and back. Could it be, that geom slows something down (in some boxes the reading ops are very slow; in my box the writing ops are very slow)? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com