From owner-freebsd-performance@FreeBSD.ORG Sun Feb 6 13:27:06 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9DD0716A4CE; Sun, 6 Feb 2005 13:27:06 +0000 (GMT) Received: from postfix4-2.free.fr (postfix4-2.free.fr [213.228.0.176]) by mx1.FreeBSD.org (Postfix) with ESMTP id 54B4F43D1D; Sun, 6 Feb 2005 13:27:06 +0000 (GMT) (envelope-from tataz@tataz.chchile.org) Received: from tatooine.tataz.chchile.org (vol75-8-82-233-239-98.fbx.proxad.net [82.233.239.98]) by postfix4-2.free.fr (Postfix) with ESMTP id 4E12C2BC3EC; Sun, 6 Feb 2005 14:27:04 +0100 (CET) Received: by tatooine.tataz.chchile.org (Postfix, from userid 1000) id DD595407C; Sun, 6 Feb 2005 14:26:42 +0100 (CET) Date: Sun, 6 Feb 2005 14:26:42 +0100 From: Jeremie Le Hen To: Robert Watson Message-ID: <20050206132642.GP163@obiwan.tataz.chchile.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.6i cc: performance@FreeBSD.org Subject: Re: Some initial postmark numbers from a dual-PIII+ATA, 4.x and 6.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2005 13:27:06 -0000 Hi Robert, > This would seem to place it closer to 4.x than 5.x -- possibly a property > of a lack of preemption. Again, the differences here are so small it's a > bit difficult to reason using them. Thanks for the result. I'm quite dubitative now : I thought this was a fact that RELENG_5 have worse performances than RELENG_4 for the moment, partly due to lack of micro-optimizations. There have been indeed numerous reports about weak performances on 5.x. Seeing your results, it appears that RELENG_4, RELENG_5 and CURRENT are in fact very close. What should we think then ? -- Jeremie Le Hen jeremie@le-hen.org From owner-freebsd-performance@FreeBSD.ORG Sun Feb 6 14:44:42 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 942FD16A4CE for ; Sun, 6 Feb 2005 14:44:42 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B11D43D3F for ; Sun, 6 Feb 2005 14:44:42 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with SMTP id A521846B11; Sun, 6 Feb 2005 09:44:41 -0500 (EST) Date: Sun, 6 Feb 2005 14:43:46 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Jeremie Le Hen In-Reply-To: <20050206132642.GP163@obiwan.tataz.chchile.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: performance@FreeBSD.org Subject: Re: Some initial postmark numbers from a dual-PIII+ATA, 4.x and 6.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2005 14:44:42 -0000 On Sun, 6 Feb 2005, Jeremie Le Hen wrote: > Hi Robert, > > > This would seem to place it closer to 4.x than 5.x -- possibly a property > > of a lack of preemption. Again, the differences here are so small it's a > > bit difficult to reason using them. > > Thanks for the result. I'm quite dubitative now : I thought this was a > fact that RELENG_5 have worse performances than RELENG_4 for the moment, > partly due to lack of micro-optimizations. There have been indeed > numerous reports about weak performances on 5.x. Seeing your results, > it appears that RELENG_4, RELENG_5 and CURRENT are in fact very close. > What should we think then ? You should think that benchmark results are a property of several factors: - Work load - Software baseline - Hardware configuration - Software configuration - Experimental method - Effectiveness of documentation Let's evaluate each: - The workload was postmark in a relatively stock configuration. I selected a smaller number of transactions than some other reporters, based on the fact that my hardware is quite a bit slower and I wanted to try and get coverage of a number of versions. I selected a 90-ish second run. The postmark benchmark is basically about effective caching, efficient I/O processes, and how the file system managed meta-data. - Software baseline: I selected to run with 4.x, 5.x, and 6.x kernels, all configured for "production" use. I.e., no debugging features enabled. I also used a statically compiled 4.x postmark binary for all tests on any versions, to try and avoid the effects of compiler changes, etc. I was primarily interested in evaluating the performance of the kernel as a variable. - Hardware configuration: I'm using somewhat dated PIII MP hardware with a relatively weak I/O path. It was the hardware on-hand and easily preemptible. The hardware has pretty good CPU:I/O performance, meaning that with many interesting workloads, the work will be I/O-bound, not CPU-bound. It becomes a question of feeding the CPUs and keeping the available I/O path used effectively. - Software configuration: I network booted the kernel, and used one of two user spaces on disk -- a 4.x world and a 6.x world. However, I used a single shared UFS1 partition for the postmark target. My hope was that static linkiing would eliminate issues involving library changes, and that using the same file system partition would help reduce disk location effects (note that disk performance varies substantially based on the location of data on the platter -- if you lay out a disk into several partitions, they will have quite different performance properties -- often in excess of the measurable experimental results of the property you're testing for). However, as a result I used UFS1 for both tests, which is not the default install configuration for FreeBSD 5.x and 6.x. - Experimental method: I attempted to control additional variables in as much as possible. However, I used a small number of runs per configuration: two. I selected that number to illustrate whether there were caching effects in play between multiple runs without reboots. The numbers suggest slight caching effects, but not huge ones. The numbers weren't large enough to give a sampling distribution that could be analyzed -- on the other hand, they were relatively long runs resulting in "mean results", meaning that we benefited from a sampling effect and a smoothing effect by virtue of the experiment design. To run this experiment properly, you'd want to distinguish the caching/non-caching cases better, control the time between runs better, and have larger samples. In order to try to explain the results I got, I waved my hands at CPU cost, and will go into that some more below. I did not test the CPU load during the experiment in a rigorous or reproduceable way. - Effectiveness of documentation: my experiment was documented, although not in great detail. I neglected to document the version of postmark (1.5c), the partition layout details, and the complete configuration details. I've included more here. In my original results post, I demonstrated that, subject to the conditions of the tests (documented above and previously), FreeBSD 5.x/6.x performance was in line with 4.x performance, or perhaps marginally faster. This surprised me also: I expected to see a 5%-10% performance drop on UP based on increased overhead, and hoped for a moderate measurable SMP performance gain relative to 4.x. On getting the results I did, I reran a couple of sample cases -- specifically, 4.x and 6.x kernels on SMP with some informal measurement of system time. I concluded that the systems were basically idle throughout the tests, which was a likely result of the I/O path being the performance bottleneck. It's likely that the slight performance improvement between 4.x and 6.x relates to preemption and the ability to turn around I/O's in the ATA driver faster, or maybe some minor pipelining effect in GEOM or such. It would be interesting to know what it is that makes 6.x faster, but it may be hard to find out given the amount of change in the system. I also informally concluded that 6.x was seeing a higher percentage system time than 4.x. This result needs to be investigated properly in an experiment of its own, since it was based on informal watching of %system in systat, combined with a subjective observation that the numbers appeared bigger. An experiment involving the use of time(1) would be a good place to start. What's interesting about this informal observation (not a formal experimental conclusion!) is that it might explain the differing postmark result from some of the other reporters. The system I tested on has a decent CPU oomph, but it's relatively slow ATA drive technology -- not a RAID, not UDMA100, etc. So if a bit more CPU was burned to get slightly more efficient use of the I/O channel, then that was immediately visible as a positive factor. On systems with much stronger I/O capabilities, perhaps to the point of being CPU-bound, that can hurt rather than help, as there are fewer resources available to support the critical path. Another point that may have helped my configuration is that it ran on a PIII, where the relative costs of synchronization primitives are much lower. A few months ago, I ran a set of micro-benchmarks that illustrated that on the P4 architecture, synchronization primitives are many times more expensive than regular operations when compared with previous architectures. It could be that the instruction blend came out "net worse" in the 5.x/6.x systems on P4-based hardware. Another point in the favor of the configuration I was running in was that the ATA driver is MPSAFE. This means its interrupt handler is able to preempt most running code, and that it can execute effectively in parallel against other parts of the kernel (including the file system). Several of the reported results were on the twe storage adapter, which does not have that property. Last night, Scott Long mailed me patches to fix dumping on twe, and also make it MPSAFE. I hope to run some stability testing on that, and then hopefully we can get those patches into the hands of people doing performance testing with twe and see if they help. FWIW, similar changes on amr and ips have resulted in substantial I/O improvements, primarily by increasing the number of transactions per second throughput by reducing latency in processing the I/O transactions. It's easy to imagine this having a direct effect on a benchmark that is really a measure of meta-data transaction throughput. Finally, my slightly hazy recollection of earlier posts was that postmark generally illustrated somewhat consistent performance between FreeBSD revisions (excepting NFS async breakage), but that Linux seemed to tromp all over on meta-data operations. There was some hypothesizing by Matt and Poul-Henning that this was a result of having what Poul-Henning refers to as a "Lemming Syncer" -- i.e., a design issue in the way we stream data to disk. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Mon Feb 7 08:36:41 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6227616A4CE for ; Mon, 7 Feb 2005 08:36:41 +0000 (GMT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id DF02943D54 for ; Mon, 7 Feb 2005 08:36:40 +0000 (GMT) (envelope-from samspeed@mail.ru) Received: from [213.129.119.20] (port=2758 helo=192.168.3.7) by mx2.mail.ru with esmtp id 1Cy4Nn-000MPg-00 for freebsd-performance@freebsd.org; Mon, 07 Feb 2005 11:36:36 +0300 Date: Mon, 7 Feb 2005 11:33:40 +0300 From: Andrey Smagin X-Mailer: The Bat! (v1.62r) X-Priority: 3 (Normal) Message-ID: <1189321068.20050207113340@mail.ru> To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam: Not detected Subject: My notices about ATA performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: samspeed List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2005 08:36:41 -0000 Hi ALL. I have 2 PC. 1. iP 166 (RAM 48Mb) 2. Duron 1133 (RAM 1Gb) 2 HDD - WD and Seagate (120GB/8Mb cache). Kernels used FreeBSD 4.8 RELEASE FreeBSD 5.1 RELEASE FreeBSD 6.0 CURRENT (1-2 week old) Test speed of copying 1 large (600Mb) file from 1 HDD to another. If put HDD's on PC 1 under 4.8 - 7-10 MBytes/s (UDMA33)(60%CPU Usage) under 5.1 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) under 6.0 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) If put better CPU K6-2-233 on PC 1 then speed of copying increased under 5.1 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) under 6.0 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) If put HDD's on PC 2 under 4.8 not tested under 5.1 - 34-41 MBytes/s (UDMA100)(~30%CPU Usage) under 6.0 - 33-41 MBytes/s (UDMA100)(~30%CPU Usage) Why speed of IO operations very depend from CPU power in 5x and 6.x ? -- Best regards, Andrey mailto:samspeed@mail.ru From owner-freebsd-performance@FreeBSD.ORG Mon Feb 7 06:02:59 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0B6116A4CE for ; Mon, 7 Feb 2005 06:02:59 +0000 (GMT) Received: from f9.mail.ru (f9.mail.ru [194.67.57.39]) by mx1.FreeBSD.org (Postfix) with ESMTP id 76A1943D5C for ; Mon, 7 Feb 2005 06:02:59 +0000 (GMT) (envelope-from samspeed_____@mail.ru) Received: from mail by f9.mail.ru with local id 1Cy1z8-000Lei-00 for freebsd-performance@freebsd.org; Mon, 07 Feb 2005 09:02:58 +0300 Received: from [213.129.119.20] by win.mail.ru with HTTP; Mon, 07 Feb 2005 09:02:58 +0300 From: Andrey Smagin To: freebsd-performance@freebsd.org Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: 192.168.3.7 via proxy [213.129.119.20] Date: Mon, 07 Feb 2005 09:02:58 +0300 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: X-Mailman-Approved-At: Mon, 07 Feb 2005 09:07:20 +0000 Subject: My notices about ATA performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Andrey Smagin List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2005 06:02:59 -0000 Hi ALL. I have 2 PC. 1. iP 166 (RAM 48Mb) 2. Duron 1133 (RAM 1Gb) 2 HDD - WD and Seagate (120GB/8Mb cache). Kernels used FreeBSD 4.8 RELEASE FreeBSD 5.1 RELEASE FreeBSD 6.0 CURRENT (1-2 week old) Test speed of copying 1 large (600Mb) file from 1 HDD to another. If put HDD's on PC 1 under 4.8 - 7-10 MBytes/s (UDMA33)(60%CPU Usage) under 5.1 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) under 6.0 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) If put better CPU K6-2-233 on PC 1 then speed of copying increased under 5.1 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) under 6.0 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) If put HDD's on PC 2 under 4.8 not tested under 5.1 - 34-41 MBytes/s (UDMA100)(~30%CPU Usage) under 6.0 - 33-41 MBytes/s (UDMA100)(~30%CPU Usage) Why speed of IO operations very depend from CPU power in 5x and 6.x ? From owner-freebsd-performance@FreeBSD.ORG Tue Feb 8 06:40:33 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD22616A4FC for ; Tue, 8 Feb 2005 06:40:32 +0000 (GMT) Received: from mailout2.pacific.net.au (mailout2.pacific.net.au [61.8.0.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2DCBB43D2D for ; Tue, 8 Feb 2005 06:40:32 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.0.86])j186eUHn029337; Tue, 8 Feb 2005 17:40:30 +1100 Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) j186eSVU019647; Tue, 8 Feb 2005 17:40:29 +1100 Date: Tue, 8 Feb 2005 17:40:27 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Andrey Smagin In-Reply-To: Message-ID: <20050208164546.W21504@delplex.bde.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-performance@FreeBSD.org Subject: Re: My notices about ATA performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2005 06:40:33 -0000 On Mon, 7 Feb 2005, Andrey Smagin wrote: > I have 2 PC. > 1. iP 166 (RAM 48Mb) > 2. Duron 1133 (RAM 1Gb) > > 2 HDD - WD and Seagate (120GB/8Mb cache). These newer drives shouldn't be the bottleneck, especially with older CPUs (and motherboards?). > Kernels used > FreeBSD 4.8 RELEASE > FreeBSD 5.1 RELEASE > FreeBSD 6.0 CURRENT (1-2 week old) > > Test speed of copying 1 large (600Mb) file > from 1 HDD to another. To test only ATA, copy between (raw) partitions. Using whole/large partitions is easiest, but the i/o size doesn't need to be nearly as large as for a file since the (FreeBSD) disk cache is not involved. > If put HDD's on PC 1 > under 4.8 - 7-10 MBytes/s (UDMA33)(60%CPU Usage) > under 5.1 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) > under 6.0 - 2.7-3 MBytes/s (UDMA33)(100%CPU Usage) 60% may be too high. 100% is too high. Check that UDMA33 is actually enabled, and where the CPU usage is. The latter is easier under 5.x+ using top -S. If the problem is interrupt overhead due to not using DMA, then it is easy to verify using systat -v. > If put better CPU K6-2-233 on PC 1 then speed of copying increased > under 5.1 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) > under 6.0 - 3.2-4 MBytes/s (UDMA33)(100%CPU Usage) I have a K6-1-233 handy for testing and noticed a drop loss of performance a year or 2 ago between RELENG_4 and RELENG_5 or possibly between ATA and ATAng on it in ATAng a year or two ago, and breakage by ATAng a little later (see below). I suspect the problem is loss of support for correct DMA timing on the old VIA motherboard on this system. The problem may be increased by the system's drive also being old and slow: % atapci0: port 0x6000-0x600f at device 7.1 on pci0 % ad0: 407MB [1002/16/52] at ata0-master WDMA0 % acd0: CDROM at ata0-slave PIO3 This drive has interesting behaviour related to DMA: - it doesn't claim to support DMA. It is so old (bought in 1995 as a cheap replacement for a failing drive?) that it doesn't support most of the feature bits in the ATA standard. However, DMA used to worked perfectly with it using the old wd driver, provided DMA was forcibly configured. IIRC, it worked OK using the ata driver too until ATAng. The above dmesg output shows that ata now decides to use DMA although the drive doesn't claim to support DMA. However, the drive stopped working in DMA starting with ATAng (DMA mode gives write errors). I work around this by putting the drive in PIO mode in /etc/rc.early: % atacontrol mode 0 PIO2 PIO3 This gives large CPU overhead instead. About 60% for reading at the slow reading at the slow rate of 1MB/sec. The drive can only do 2MB/sec at best and gets close to that in DMA mode. Your newer drives shouldn't have this problem, but the old motherboard might. > If put HDD's on PC 2 > under 4.8 not tested > under 5.1 - 34-41 MBytes/s (UDMA100)(~30%CPU Usage) > under 6.0 - 33-41 MBytes/s (UDMA100)(~30%CPU Usage) This is close to normal. DMA mode takes very little CPU. The slower CPUs are overloaded by 50MB/sec disks for copying the data to and from userland (a K6-1-233 can only copy at 40MB/sec), but they shouldn't have 100% cpu usage for file copies at only 2.7-4MB/sec. > Why speed of IO operations very depend from CPU power in 5x and 6.x ? 6.x has more pessimizations. However, these should have only small effects for disk i/o and other i/o operations that involve large (larger than a few KB) buffers. Bruce From owner-freebsd-performance@FreeBSD.ORG Tue Feb 8 07:21:33 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E7EE16A4CE for ; Tue, 8 Feb 2005 07:21:33 +0000 (GMT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id E225443D1D for ; Tue, 8 Feb 2005 07:21:32 +0000 (GMT) (envelope-from samspeedu@mail.ru) Received: from [213.129.119.20] (port=1360 helo=192.168.3.7) by mx2.mail.ru with esmtp id 1CyPgg-000OKu-00; Tue, 08 Feb 2005 10:21:31 +0300 Date: Tue, 8 Feb 2005 10:18:32 +0300 From: Andrey Smagin X-Mailer: The Bat! (v1.62r) Organization: DiP X-Priority: 3 (Normal) Message-ID: <638754843.20050208101832@mail.ru> To: Bruce Evans In-Reply-To: <20050208164546.W21504@delplex.bde.org> References: <20050208164546.W21504@delplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam: Not detected X-Mailman-Approved-At: Tue, 08 Feb 2005 07:30:13 +0000 cc: freebsd-performance@FreeBSD.org Subject: Re: My notices about ATA performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: SAMU List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2005 07:21:33 -0000 Ok. I tried to increase speed of IO operations by setting Promise ATA100 controller, both IDE drives worked in ATA100 mode but speed also was fully CPU depend and 2.7-3MB/s. Also copying in network or /dev/null made same speed. HDD was connected in different channels of controller. Using 6.0 current not give me more performance. My kernel configuration is GENERIC with some USB devices (ucom,umodem...), without IPv6 and some unused NIC drivers. Also i seen DMA-READ errors with 5.x and 6.x with HDD, but with old 4.x it never took place. -- Best regards, Andrey mailto:samspeedu@mail.ru From owner-freebsd-performance@FreeBSD.ORG Wed Feb 9 21:03:36 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3E34C16A4CE for ; Wed, 9 Feb 2005 21:03:36 +0000 (GMT) Received: from gigave.com (mail.gigave.com [38.113.228.9]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0FC9143D1D for ; Wed, 9 Feb 2005 21:03:36 +0000 (GMT) (envelope-from sean@sean.gigave.com) Received: from sean.gigave.com [38.113.228.242](yr5ohb0hvaukpquj)by gigave.comwith esmtp(Exim 4.43 #1 (Gentoo Linux))id 1Cyz3m-0006S8-DX; Wed, 09 Feb 2005 13:07:42 -0800 Received: by sean.gigave.com (Postfix, from userid 1001) id 53DFE11B55; Wed, 9 Feb 2005 13:03:35 -0800 (PST) Date: Wed, 9 Feb 2005 13:03:35 -0800 From: Sean Chittenden To: performance@FreeBSD.org Message-ID: <20050209210335.GE1060@sean.gigave.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.6i X-Mailman-Approved-At: Wed, 09 Feb 2005 21:12:11 +0000 Subject: [Success Story] amd64 + squid + apache + MySQL... X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2005 21:03:36 -0000 Hey, just thought I'd pass along a quick note for a 5.X/amd64 success story. Given those who have had success with this haven't spoken up or said as much publicly, let me buck the trend and spread some good news to those interested in listening. Athlon64 3000 (UP), em0, 2GB of RAM, 3Ware 9000 SATA RAID controller Squid 2.5.7, Apache/mod_php 1.3, MySQL 4.1 The above is yielding (at peak times) ~85Mbps with ~5K connections.... or more ('ya got to love fluctuating real numbers): # netstat -na | grep ESTABLISHED | wc -l 6783 # netstat -I em0 -w1 | head -3 input (em0) output packets errs bytes packets errs bytes colls 6968 0 2060646 9686 0 11175649 0 # w 12:44PM up 1 day, 17:05, 1 user, load averages: 3.67, 3.66, 3.29 USER TTY FROM LOGIN@ IDLE WHAT sean p0 foo.example.com 8:21AM - w So there 'ya go: a real world amd64 and RELENG_5 success story. I know there are other folks out there doing more, but, still... what should be a reasonably impressive data point. From our perspective, more hardware is needed for redundancy's sake, but, make no mistake, we couldn't be more pleased with the end results and the process by which we got there (my only gripe was the inability to update the firmware for the 3ware RAID card, for some reason I couldn't flash the controller, though it's not too critical, IMHO. It's one of those, "it would've been nice to have done before going live" kinda things). I will say, however, that we shocked the hell out of the client when we told them that post-move, they were pushing 2-3x their earlier traffic because we weren't capping them and/or they have a higher ceiling with the new hardware (we spec'ed the hardware for 30Mbps). Closer to the advocacy/trolling side (please, do not, under any circumstances reply to the list regarding this unless you have a success story of your own! If you fail this test, I will moderate future postings by you to this list), this particular client was running on a 10 machine Linux Virtual Server setup and was tipping frequently. I don't mean to suggest that this can always be done (I know it can't, but I also don't want to deter people from trying), but, the steady hand of FreeBSD under load has been immensely impressive to all parties involved on this side who watched it happen. Blindly turning up an amd64+RELENG_5 system without any load testing in advance... very convincing to watch happen, IMHO. Off list, if someone wants to give me pointers for getting squid 3.0 compiled/running (PRE3 wouldn't compile), or has any good/bad news on its kqueue(2) support, I'm all ears. select(2) is starting to be a problem under higher connection counts and is adding unnecessary latency. I'm be curious to know if anyone has made or is aware of a thread safe select(3) function that uses kqueue(2) on the backend. Something that can be dynamically loaded as a shared object which is invoked in place of select(2) and shoehorned in for legacy software. FreeBSD. Silent workhorse no more. -sc -- Sean Chittenden sean@GigAve.com