From owner-freebsd-performance@FreeBSD.ORG Sun Aug 9 04:06:55 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BE40106564A for ; Sun, 9 Aug 2009 04:06:55 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-173-50-131-130.ptldor.fios.verizon.net [173.50.131.130]) by mx1.freebsd.org (Postfix) with ESMTP id 7E5BB8FC0A for ; Sun, 9 Aug 2009 04:06:53 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id C7B4CB64F; Sat, 8 Aug 2009 20:36:45 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id EAA08610; Sun, 9 Aug 2009 04:02:43 GMT Message-Id: <200908090402.EAA08610@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Sat, 08 Aug 2009 05:02:48 EDT." Date: Sat, 08 Aug 2009 21:02:43 PDT From: Dieter Subject: Re: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2009 04:06:55 -0000 > I can dd if=/dev/ad[n].eli of=/dev/null bs=1m and use 75% system > all in geli, 27% disk busy, 20MiB/sec. Interface was slower but > reasonable. I think I understand now. You're doing encryption in the kernel, which eats a lot of cpu, and nice only affects userland. So yeah cpu is a significant part of your problem. > I'm not sure yet how to isolate cpu from i/o under my geli+zfs > setup. I think they're mated together. Agreed. > It's just that this workload has really put the screws to things > and I don't see a way out. I'd like to deploy geli+zfs everywhere > but if I can't login remotely because some user has it busied out > on something I've no knobs to control, umm, yeah :) Do you *need* geli+zfs ? If so, you could see if there are any hardware crypto accellerators with FreeBSD support, or throw lots of cpu (e.g. phenom2 x4) at it. > As to your i/o thing, I think back in RELENG_4 that if all the > spindles were on the same pata controller/interrupt, monopolistic > loads could occur. atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe000-0xe00f at device 6.0 on pci0 atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xcc00-0xcc0f mem 0xfebfb000-0xfebfbfff irq 21 at device 7.0 on pci0 atapci2: port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xb800-0xb80f mem 0xfebfa000-0xfebfafff irq 22 at device 8.0 on pci0 atapci3: port 0x8c00-0x8c07,0x8800-0x8803,0x8400-0x8407,0x8000-0x8003,0x7c00-0x7c0f mem 0xfe9fe000-0xfe9fffff irq 17 at device 0.0 on pci3 atapci4: port 0x6c00-0x6c7f mem 0xfe6ff000-0xfe6ff07f,0xfe6f8000-0xfe6fbfff irq 16 at device 0.0 on pci4 atapci5: port 0x4c00-0x4c07,0x4800-0x4803,0x4400-0x4407,0x4000-0x4003,0x3c00-0x3c0f mem 0xfe3fe000-0xfe3fffff irq 18 at device 0.0 on pci6 The nForce pata controller doesn't list an irq, seems odd? From owner-freebsd-performance@FreeBSD.ORG Mon Aug 10 08:47:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 221C1106566C for ; Mon, 10 Aug 2009 08:47:18 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx44.mail.ru (mx44.mail.ru [94.100.176.58]) by mx1.freebsd.org (Postfix) with ESMTP id 9D3778FC30 for ; Mon, 10 Aug 2009 08:47:17 +0000 (UTC) Received: from mx38.mail.ru (mx38.mail.ru [94.100.176.52]) by mx44.mail.ru (mPOP.Fallback_MX) with ESMTP id 5D67C30AE56 for ; Mon, 10 Aug 2009 12:30:07 +0400 (MSD) Received: from [91.190.115.253] (port=59538 helo=pstation) by mx38.mail.ru with asmtp id 1MaQGX-0008xL-00; Mon, 10 Aug 2009 12:30:01 +0400 Date: Mon, 10 Aug 2009 12:32:27 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <117233873703.20090810123227@mail.ru> To: Invernizzi Fabrizio In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam: Not detected X-Mras: Ok Cc: freebsd-performance@freebsd.org Subject: Re[2]: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2009 08:47:18 -0000 Hello Invernizzi, First of all I didn't catch for what task you are tuning. For forwarding or for transfer? In any case you should start with minimal configuration (no netgraph modules, no any firewall). Choose a test similiar to your typical load. Provide us with test result, cpu consumption and output of ifconfig netstat -s netstat -m debug values from driver (sysctl (?).debug)). Wednesday, August 05, 2009, 2:13:23 PM, you wrote: IF> No improvement with kern.ipc.nmbclusters=262144 and 1.8.6 driver :<((((( IF> ++fabrizio IF> ------------------------------------------------------------------ IF> Telecom Italia IF> Fabrizio INVERNIZZI IF> Technology - TILAB IF> Accesso Fisso e Trasporto IF> Via Reiss Romoli, 274 10148 Torino IF> Tel. +39 011 2285497 IF> Mob. +39 3316001344 IF> Fax +39 06 41867287 IF> ________________________________ IF> From: Jack Vogel [mailto:jfvogel@gmail.com] IF> Sent: marted́ 4 agosto 2009 18.42 IF> To: Julian Elischer IF> Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev IF> Subject: Re: Test on 10GBE Intel based network card IF> Your nmbclusters is very low, you list it twice so I'm assuming the second value is IF> what it ends up being, 32K :( IF> I would set it to: IF> kern.ipc.nmbclusters=262144 IF> Also, I thought you were using the current driver, but now it looks like you are IF> using something fairly old, use my latest code which is 1.8.8 IF> Jack IF> On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: IF> Invernizzi Fabrizio wrote: IF> The limitation that you see is about the max number of packets that IF> FreeBSD can handle - it looks like your best performance is reached at IF> 64 byte packets? IF> If you are meaning in term of Packet per second, you are right. These IF> are the packet per second measured during tests: IF> 64 byte: 610119 Pps IF> 512 byte: 516917 Pps IF> 1492 byte: 464962 Pps IF> Am I correct that the maximum you can reach is around 639,000 packets IF> per second? IF> Yes, as you can see the maximum is 610119 Pps. IF> Where does this limit come from? IF> ah that's the whole point of tuning :-) IF> there are severalpossibities: IF> 1/ the card's interrupts are probably attache dto aonly 1 cpu, IF> so that cpu can do no more work IF> This seems not to be the problem. See below a top snapshot during a 64byte-long packet storm IF> last pid: 8552; load averages: 0.40, 0.09, 0.03 up 0+20:36:58 09:40:29 IF> 124 processes: 13 running, 73 sleeping, 38 waiting IF> CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle IF> Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free IF> Swap: 2048M Total, 2048M Free IF> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND IF> 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: cpu3 IF> 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: cpu0 IF> 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: cpu2 IF> 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: cpu1 IF> 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq IF> 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq IF> 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq IF> 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq IF> .... IF> It looks like the 4 rxq processes are bound to the 4 available cores with equal distribution. IF> 2/ if more than 1 cpu is working, it may be that there is a lock in IF> heavy contention somewhere. IF> This I think is the problem. I am trying to understand how to IF> 1- see where the heavy contention is (context switching? Some limiting setting?) IF> 2- mitigate it IF> there ia a lock profiling tool that right now I can't remember the name of.. IF> look it up with google :-) FreeBSD lock profiling tool IF> ah, first hit... IF> http://blogs.epfl.ch/article/23832 IF> is the machine still responsive to other networks while IF> running at maximum capacity on this network? (make sure that IF> the other networks are on a differnet CPU (hmm I can't remember how to IF> do that :-). -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-performance@FreeBSD.ORG Mon Aug 10 23:53:54 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 061E51065673 for ; Mon, 10 Aug 2009 23:53:54 +0000 (UTC) (envelope-from nathan@lenevez.net.au) Received: from smtp1.l33t.net.au (smtp1.l33t.net.au [IPv6:2400:5000:1337:a::71]) by mx1.freebsd.org (Postfix) with ESMTP id 899B28FC25 for ; Mon, 10 Aug 2009 23:53:53 +0000 (UTC) Received: from talon.lenevez.net.au (smtp-outbound.lenevez.net.au [119.15.98.178]) by smtp1.l33t.net.au (8.14.2/8.14.2) with ESMTP id n7ANroj3034475 for ; Tue, 11 Aug 2009 09:53:50 +1000 (EST) (envelope-from nathan@lenevez.net.au) Received: from talon.lenevez.net.au ([119.15.98.178]) by talon.lenevez.net.au ([119.15.98.178]) with mapi; Tue, 11 Aug 2009 09:52:14 +1000 From: Nathan Le Nevez To: "freebsd-performance@freebsd.org" Date: Tue, 11 Aug 2009 09:53:47 +1000 Thread-Topic: Very slow I/O performance on HP BL465c Thread-Index: AQHKGhU4FOVAZZOuqE2TtD8XIPHZGZCf92/Y Message-ID: In-Reply-To: Accept-Language: en-US, en-AU Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-AU Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.0 (smtp1.l33t.net.au [119.15.98.71]); Tue, 11 Aug 2009 09:53:50 +1000 (EST) X-L33T-MailScanner-Information: Please contact the ISP for more information X-MailScanner-ID: n7ANroj3034475 X-L33T-MailScanner: Found to be clean X-L33T-MailScanner-From: nathan@lenevez.net.au X-Spam-Status: No Subject: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2009 23:53:54 -0000 Hi, I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which performs ver= y poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks configured in RAID-1. Both controllers have write-cache enabled. Both servers are running the same BIOS and firmware versions. Neither servers ar= e running any services other than sshd. Blade with good performance (2 x Opteron 2218, 8GB RAM): ciss0: port 0x4000-0x40ff mem 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 ciss0: [ITHREAD] da0 at ciss0 bus 0 target 0 lun 0 da0: AFPi xCePdU D#i2r Leacuntc hAcecde!s s SCSI-5 device da0: 135.168MB/s transfers dSaM0P:: CAoP CPU #3 Launched! mmand Queueing Enabled da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) Blade with bad performance (2 x Opteron 2352, 16GB RAM): ciss0: port 0x4000-0x40ff mem 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 ciss0: [ITHREAD] da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing Enabled da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) # dbench -t 10 1 2 3 4 blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/sec blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/sec Any help/ideas would be greatly appreciated. I have run through all the Insight diagnostics tools and it fails to find anything wrong with the slow server. Cheers, Nathan=20 From owner-freebsd-performance@FreeBSD.ORG Tue Aug 11 03:03:01 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9844A106564A for ; Tue, 11 Aug 2009 03:03:01 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from ey-out-2122.google.com (ey-out-2122.google.com [74.125.78.27]) by mx1.freebsd.org (Postfix) with ESMTP id 2FB1E8FC1F for ; Tue, 11 Aug 2009 03:03:00 +0000 (UTC) Received: by ey-out-2122.google.com with SMTP id 9so793899eyd.7 for ; Mon, 10 Aug 2009 20:03:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=CoHQ0HqvXHSfHi3RDvMqn/vvARg9cYaLoyofZmdz3js=; b=YYAmooqFYX9hKEUSCA9TdqIkG/ecrgD54u9QKr+WtoNthNl+K1l8rrCea0aRGzKZxG Jd4mxkTW4UXtxBszNpAUeR4xhv0SXDPAQzaxvyVwFYxfi4bIE7NC1DHbHdHRqvjU0D5Q hi502nA0yFVV+RbB7zrV+zdH1/cl5KvDaRXMU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=BP1LwrN7DaOvK9TVNj9pgy3aEW3wAh0daPdrA+b+VnQiKKBtVMt/UZfctmnDuIDkKe TEg7WSXzpIuMUo9L1uaQg18Q2iJ2fOpzmZ5qyDdNewEyYaE9dwHqpVY4tIrwjYgEyw8l r+DRVY1c/lovRmBthv+cimiD2jQpCKbPEsw0c= MIME-Version: 1.0 Received: by 10.210.111.5 with SMTP id j5mr5786460ebc.55.1249959780189; Mon, 10 Aug 2009 20:03:00 -0700 (PDT) Date: Mon, 10 Aug 2009 23:03:00 -0400 Message-ID: From: grarpamp To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 11 Aug 2009 03:19:18 +0000 Subject: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2009 03:03:01 -0000 > nice only affects userland Well, you can set {id,rt}prio and nice on kernel processes. Then look at top to see the nice column change. Have no idea what effect it has nor what the non '-' chars on those procs in that column mean. > Do you *need* geli+zfs? Encryption = required. ZFS... well I like the checksum all the way back to the uberblock feature, raidz2, ditto blocks, compression and the admin model. geom offers encryption, single block checksum authority and raid1/3. > hardware crypto accellerators The soekris ones work and are cheap. I thougt I saw posts that show openssl -speed on today's fast cpu's being faster than the accel cards. Disk crypto is symmetric, not initial pki session setup. And with ~40MB/s of encryption, while building world no less, cpu may not be the issue. So long as I'm not hitting those geli+zfs disks, things are smooth. I want to try the system with geli+ufs2 sometime. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128 cbc 36555.26k 38074.95k 38999.93k 39258.38k 39176.80k > throw lots of cpu (e.g. phenom2 x4) at it That would only help it get done busying out the system sooner, not balance things out while actively under load. Which as a file server, it always is. Well, ok, it will help after the system is able to do spindle -> geli -> fs -> process at the max sustained read/write speed of the spindles. Which is about 56MiB/sec reading in this case. Which is over 10x faster than I'm getting now. Which means I'd need maybe 10 x 1.8GHz worth of cpu before I have any free cycles to devote to the user interface :) Maybe I'm just clueless this month and the list is too busy to beat me about the head with it. > The nForce pata controller doesn't list an irq, seems odd? > device 6.0 on pci0 This one doesn't either: atapci1: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 31.1 on pci0 boot -v and it appears in the irq routing stuff around that line. Then vmstat -i and systat -vmstat 1 also give some clues when device is dd'd. Onboard PATA is always irq14/15 as I've seen. From owner-freebsd-performance@FreeBSD.ORG Tue Aug 11 06:47:54 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28F7F106566B for ; Tue, 11 Aug 2009 06:47:54 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-173-50-131-130.ptldor.fios.verizon.net [173.50.131.130]) by mx1.freebsd.org (Postfix) with ESMTP id F1E8E8FC35 for ; Tue, 11 Aug 2009 06:47:52 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id 9E416B64F; Mon, 10 Aug 2009 23:16:16 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id GAA06250; Tue, 11 Aug 2009 06:37:02 GMT Message-Id: <200908110637.GAA06250@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Mon, 10 Aug 2009 23:03:00 EDT." Date: Mon, 10 Aug 2009 23:37:02 PDT From: Dieter Subject: Re: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2009 06:47:54 -0000 > > hardware crypto accellerators > > The soekris ones work and are cheap. I thougt I saw posts that show > openssl -speed on today's fast cpu's being faster than the accel > cards. Disk crypto is symmetric, not initial pki session setup. The accel cards probably don't get updated as often as cpus do. Then there is the question of how many cards would you need and having enough free slots to plug them into. And the cards aren't going to help with zfs. Does anyone make a disk controller with crypto built in? Then it would scale with the number of drives. And not need extra slots. > > throw lots of cpu (e.g. phenom2 x4) at it > > That would only help it get done busying out the system sooner, not > balance things out while actively under load. Which as a file server, > it always is. > > Well, ok, it will help after the system is able to do spindle -> > geli -> fs -> process at the max sustained read/write speed of the > spindles. Which is about 56MiB/sec reading in this case. Which is > over 10x faster than I'm getting now. Which means I'd need maybe > 10 x 1.8GHz worth of cpu before I have any free cycles to devote > to the user interface :) Well, you could look into mainboards with 2 or 4 CPU sockets. Tyan probably has something. > Maybe I'm just clueless this month and the list is too busy to beat > me about the head with it. Anyone running geli+zfs is sitting there waiting for the user interface to update their screen. :-) >>> The data disks are hanging off a dumb ata133 pdc20269 card Ok, here is a REALLY UGLY way to slow the disks down: for disk in ad2 ad4 ... do atacontrol mode disk UDMA66 done Not a good solution. If one transfer is eating your user interface, with 8 drives you'd have to set the speed very very low. You really want a way to set the priority. BTW, if your data is so important that encryption is required, "checksum all the way back to the uberblock", etc. etc. you might want to upgrade the disks from pata to at least sata. Pata doesn't do error detection on the control info, so it could happily write your bits to the wrong sector. Sata checks the control info as well as the data. Current 7200 rpm sata speeds: dd reading the bare drive, no filesystem, at fast end of the drive: extended device statistics device r/s w/s kr/s kw/s wait svc_t %b ad18 1871.1 0.0 118303.1 0.0 2 0.8 98 dd reading a file from FFS: 28125720000 bytes transferred in 264.921699 secs (106166162 bytes/sec) Or you could go with the green disks that spin slower. From owner-freebsd-performance@FreeBSD.ORG Tue Aug 11 12:09:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FADD106566B for ; Tue, 11 Aug 2009 12:09:18 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 044958FC20 for ; Tue, 11 Aug 2009 12:09:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id 8F3E7C3BF; Tue, 11 Aug 2009 14:44:43 +0300 (EEST) Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 79530-02; Tue, 11 Aug 2009 14:44:42 +0300 (EEST) Received: from [192.168.2.188] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id B1E2AC3B7; Tue, 11 Aug 2009 14:44:42 +0300 (EEST) Message-ID: <4A8159A8.3030206@bulinfo.net> Date: Tue, 11 Aug 2009 14:44:40 +0300 From: Krassimir Slavchev User-Agent: Thunderbird 2.0.0.19 (X11/20090225) MIME-Version: 1.0 To: Nathan Le Nevez References: In-Reply-To: X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at mx.bulinfo.net Cc: "freebsd-performance@freebsd.org" Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2009 12:09:18 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, What is the output of 'vmstat -i' and 'camcontrol tags da0' ? I have a ML350 running 7-STABLE with same controller and disks and performance is almost same as your good server. Nathan Le Nevez wrote: > Hi, > > I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which performs very > poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks > configured in RAID-1. Both controllers have write-cache enabled. Both > servers are running the same BIOS and firmware versions. Neither servers are > running any services other than sshd. > > Blade with good performance (2 x Opteron 2218, 8GB RAM): > > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: AFPi xCePdU D#i2r Leacuntc hAcecde!s > s SCSI-5 device > da0: 135.168MB/s transfers > dSaM0P:: CAoP CPU #3 Launched! > mmand Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > > Blade with bad performance (2 x Opteron 2352, 16GB RAM): > > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 135.168MB/s transfers > da0: Command Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > > # dbench -t 10 1 2 3 4 > blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/sec > blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/sec > > Any help/ideas would be greatly appreciated. I have run through all the > Insight diagnostics tools and it fails to find anything wrong with the slow > server. > > Cheers, > Nathan > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFKgVmoxJBWvpalMpkRAvilAJsGF0J34SgD34EcBxX8Ic8Hq6OUBACghpBL C7YgX2qmvgb7WSvgFhDrKl8= =JDJx -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Tue Aug 11 21:29:07 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79D65106566B for ; Tue, 11 Aug 2009 21:29:07 +0000 (UTC) (envelope-from nathan@lenevez.net.au) Received: from smtp1.l33t.net.au (smtp1.l33t.net.au [IPv6:2400:5000:1337:a::71]) by mx1.freebsd.org (Postfix) with ESMTP id 07A9B8FC3D for ; Tue, 11 Aug 2009 21:29:06 +0000 (UTC) Received: from talon.lenevez.net.au (smtp-outbound.lenevez.net.au [119.15.98.178]) by smtp1.l33t.net.au (8.14.2/8.14.2) with ESMTP id n7BLSkTk052866; Wed, 12 Aug 2009 07:28:47 +1000 (EST) (envelope-from nathan@lenevez.net.au) Received: from talon.lenevez.net.au ([119.15.98.178]) by talon.lenevez.net.au ([119.15.98.178]) with mapi; Wed, 12 Aug 2009 07:27:11 +1000 From: Nathan Le Nevez To: Krassimir Slavchev Date: Wed, 12 Aug 2009 07:28:53 +1000 Thread-Topic: Very slow I/O performance on HP BL465c Thread-Index: AcoaeOv11wNZ5eR6TZyJcc2MshjO8wAUZK9g Message-ID: References: <4A8159A8.3030206@bulinfo.net> In-Reply-To: <4A8159A8.3030206@bulinfo.net> Accept-Language: en-US, en-AU Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-AU Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.0 (smtp1.l33t.net.au [119.15.98.71]); Wed, 12 Aug 2009 07:28:47 +1000 (EST) X-L33T-MailScanner-Information: Please contact the ISP for more information X-MailScanner-ID: n7BLSkTk052866 X-L33T-MailScanner: Found to be clean X-L33T-MailScanner-From: nathan@lenevez.net.au X-Spam-Status: No Cc: "freebsd-performance@freebsd.org" Subject: RE: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2009 21:29:07 -0000 # vmstat -i interrupt total rate irq1: atkbd0 18 0 irq5: ohci0 ohci1+ 1 0 irq19: ciss0 144916 3 irq21: uhci0 22 0 cpu0: timer 80002970 1999 irq256: bce0 17042 0 cpu2: timer 79994902 1999 cpu1: timer 79994975 1999 cpu3: timer 79995009 1999 cpu6: timer 79994957 1999 cpu5: timer 79995046 1999 cpu4: timer 79995041 1999 cpu7: timer 79995057 1999 Total 640129956 16000 # camcontrol tags da0 (pass0:ciss0:0:0:0): device openings: 254 Just for clarification, both systems are running amd64. Thanks, Nathan -----Original Message----- From: Krassimir Slavchev [mailto:krassi@bulinfo.net]=20 Sent: Tuesday, 11 August 2009 9:45 PM To: Nathan Le Nevez Cc: freebsd-performance@freebsd.org Subject: Re: Very slow I/O performance on HP BL465c -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, What is the output of 'vmstat -i' and 'camcontrol tags da0' ? I have a ML350 running 7-STABLE with same controller and disks and performance is almost same as your good server. Nathan Le Nevez wrote: > Hi, >=20 > I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which performs v= ery > poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks > configured in RAID-1. Both controllers have write-cache enabled. Both > servers are running the same BIOS and firmware versions. Neither servers = are > running any services other than sshd. >=20 > Blade with good performance (2 x Opteron 2218, 8GB RAM): >=20 > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: AFPi xCePdU D#i2r Leacuntc hAcecde!= s > s SCSI-5 device > da0: 135.168MB/s transfers > dSaM0P:: CAoP CPU #3 Launched! > mmand Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) >=20 > Blade with bad performance (2 x Opteron 2352, 16GB RAM): >=20 > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 135.168MB/s transfers > da0: Command Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) >=20 > # dbench -t 10 1 2 3 4 > blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/s= ec > blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/se= c >=20 > Any help/ideas would be greatly appreciated. I have run through all the > Insight diagnostics tools and it fails to find anything wrong with the sl= ow > server. >=20 > Cheers, > Nathan=20 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" >=20 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFKgVmoxJBWvpalMpkRAvilAJsGF0J34SgD34EcBxX8Ic8Hq6OUBACghpBL C7YgX2qmvgb7WSvgFhDrKl8=3D =3DJDJx -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Wed Aug 12 05:41:23 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27E831065672 for ; Wed, 12 Aug 2009 05:41:23 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 994BD8FC41 for ; Wed, 12 Aug 2009 05:41:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id 8BA54C4BE; Wed, 12 Aug 2009 08:41:20 +0300 (EEST) Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 13099-07; Wed, 12 Aug 2009 08:41:19 +0300 (EEST) Received: from [192.168.2.188] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id 44E5BBDA7; Wed, 12 Aug 2009 08:41:19 +0300 (EEST) Message-ID: <4A8255FE.5030500@bulinfo.net> Date: Wed, 12 Aug 2009 08:41:18 +0300 From: Krassimir Slavchev User-Agent: Thunderbird 2.0.0.19 (X11/20090225) MIME-Version: 1.0 To: Nathan Le Nevez References: <4A8159A8.3030206@bulinfo.net> In-Reply-To: X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at mx.bulinfo.net Cc: "freebsd-performance@freebsd.org" Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 05:41:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Looks okay. How your disks are partitioned and from where you are running dbench. Look at the -D option. For example I have: / without soft updates dbench -t 10 4 -> Throughput 72.7276 MB/sec 4 procs /var with soft updates -> Throughput 286.528 MB/sec 4 procs Are you sure that you are not running dbench on zfs or encrypted partition? Nathan Le Nevez wrote: > # vmstat -i > interrupt total rate > irq1: atkbd0 18 0 > irq5: ohci0 ohci1+ 1 0 > irq19: ciss0 144916 3 > irq21: uhci0 22 0 > cpu0: timer 80002970 1999 > irq256: bce0 17042 0 > cpu2: timer 79994902 1999 > cpu1: timer 79994975 1999 > cpu3: timer 79995009 1999 > cpu6: timer 79994957 1999 > cpu5: timer 79995046 1999 > cpu4: timer 79995041 1999 > cpu7: timer 79995057 1999 > Total 640129956 16000 > > # camcontrol tags da0 > (pass0:ciss0:0:0:0): device openings: 254 > > Just for clarification, both systems are running amd64. > > Thanks, > > Nathan > > -----Original Message----- > From: Krassimir Slavchev [mailto:krassi@bulinfo.net] > Sent: Tuesday, 11 August 2009 9:45 PM > To: Nathan Le Nevez > Cc: freebsd-performance@freebsd.org > Subject: Re: Very slow I/O performance on HP BL465c > > Hi, > > What is the output of 'vmstat -i' and 'camcontrol tags da0' ? > I have a ML350 running 7-STABLE with same controller and disks and > performance is almost same as your good server. > > Nathan Le Nevez wrote: >> Hi, > >> I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which performs very >> poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks >> configured in RAID-1. Both controllers have write-cache enabled. Both >> servers are running the same BIOS and firmware versions. Neither servers are >> running any services other than sshd. > >> Blade with good performance (2 x Opteron 2218, 8GB RAM): > >> ciss0: port 0x4000-0x40ff mem >> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 >> ciss0: [ITHREAD] >> da0 at ciss0 bus 0 target 0 lun 0 >> da0: AFPi xCePdU D#i2r Leacuntc hAcecde!s >> s SCSI-5 device >> da0: 135.168MB/s transfers >> dSaM0P:: CAoP CPU #3 Launched! >> mmand Queueing Enabled >> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >> Blade with bad performance (2 x Opteron 2352, 16GB RAM): > >> ciss0: port 0x4000-0x40ff mem >> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 >> ciss0: [ITHREAD] >> da0 at ciss0 bus 0 target 0 lun 0 >> da0: Fixed Direct Access SCSI-5 device >> da0: 135.168MB/s transfers >> da0: Command Queueing Enabled >> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >> # dbench -t 10 1 2 3 4 >> blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/sec >> blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/sec > >> Any help/ideas would be greatly appreciated. I have run through all the >> Insight diagnostics tools and it fails to find anything wrong with the slow >> server. > >> Cheers, >> Nathan > >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFKglX+xJBWvpalMpkRAjxzAKCpNTFtnrA3GgJn66OPB3hwTAL1FgCdFPJf 5wxj8jV4jw/Hf6sYEwXUOEo= =9Inl -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Wed Aug 12 07:57:55 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFE12106566B for ; Wed, 12 Aug 2009 07:57:55 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 55B548FC44 for ; Wed, 12 Aug 2009 07:57:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id C593EC521; Wed, 12 Aug 2009 10:57:53 +0300 (EEST) Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 17814-10; Wed, 12 Aug 2009 10:57:52 +0300 (EEST) Received: from [192.168.2.188] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id 90529C49F; Wed, 12 Aug 2009 10:57:52 +0300 (EEST) Message-ID: <4A8275FF.2000201@bulinfo.net> Date: Wed, 12 Aug 2009 10:57:51 +0300 From: Krassimir Slavchev User-Agent: Thunderbird 2.0.0.19 (X11/20090225) MIME-Version: 1.0 To: Nathan Le Nevez References: In-Reply-To: X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at mx.bulinfo.net Cc: "freebsd-performance@freebsd.org" Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 07:57:56 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Is it possible to exchange disks between your blade1 and blade2 servers? Or to remove disks from one server and connect them to another? Also compare 'tunefs -p /' outputs Also compare the read speed of a raw device with e.g. 'dd if=/dev/da0 of=/dev/null bs=1m count=100' Nathan Le Nevez wrote: > # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/da0s1a 496M 224M 232M 49% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/da0s1e 496M 14K 456M 0% /tmp > /dev/da0s1f 119G 623M 109G 0% /usr > /dev/da0s1d 4.8G 346K 4.4G 0% /var > # mount > /dev/da0s1a on / (ufs, local) > devfs on /dev (devfs, local) > /dev/da0s1e on /tmp (ufs, local, soft-updates) > /dev/da0s1f on /usr (ufs, local, soft-updates) > /dev/da0s1d on /var (ufs, local, soft-updates) > > / - Throughput 6.59862 MB/sec 4 procs > /usr - Throughput 14.487 MB/sec 4 procs > > > > On 12/08/09 3:41 PM, "Krassimir Slavchev" wrote: > > Looks okay. > How your disks are partitioned and from where you are running dbench. > Look at the -D option. For example I have: > / without soft updates -> Throughput 72.7276 MB/sec 4 > procs > /var with soft updates -> Throughput 286.528 MB/sec 4 procs > > Are you sure that you are not running dbench on zfs or encrypted > partition? > > Nathan Le Nevez wrote: >> # vmstat -i >> interrupt total rate >> irq1: atkbd0 18 0 >> irq5: ohci0 ohci1+ 1 0 >> irq19: ciss0 144916 3 >> irq21: uhci0 22 0 >> cpu0: timer 80002970 1999 >> irq256: bce0 17042 0 >> cpu2: timer 79994902 1999 >> cpu1: timer 79994975 1999 >> cpu3: timer 79995009 1999 >> cpu6: timer 79994957 1999 >> cpu5: timer 79995046 1999 >> cpu4: timer 79995041 1999 >> cpu7: timer 79995057 1999 >> Total 640129956 16000 > >> # camcontrol tags da0 >> (pass0:ciss0:0:0:0): device openings: 254 > >> Just for clarification, both systems are running amd64. > >> Thanks, > >> Nathan > >> -----Original Message----- >> From: Krassimir Slavchev [mailto:krassi@bulinfo.net] >> Sent: Tuesday, 11 August 2009 9:45 PM >> To: Nathan Le Nevez >> Cc: freebsd-performance@freebsd.org >> Subject: Re: Very slow I/O performance on HP BL465c > >> Hi, > >> What is the output of 'vmstat -i' and 'camcontrol tags da0' ? >> I have a ML350 running 7-STABLE with same controller and disks and >> performance is almost same as your good server. > >> Nathan Le Nevez wrote: >>> Hi, > >>> I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which > performs very >>> poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks >>> configured in RAID-1. Both controllers have write-cache enabled. Both >>> servers are running the same BIOS and firmware versions. Neither > servers are >>> running any services other than sshd. > >>> Blade with good performance (2 x Opteron 2218, 8GB RAM): > >>> ciss0: port 0x4000-0x40ff mem >>> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 > on pci80 >>> ciss0: [ITHREAD] >>> da0 at ciss0 bus 0 target 0 lun 0 >>> da0: AFPi xCePdU D#i2r Leacuntc > hAcecde!s >>> s SCSI-5 device >>> da0: 135.168MB/s transfers >>> dSaM0P:: CAoP CPU #3 Launched! >>> mmand Queueing Enabled >>> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >>> Blade with bad performance (2 x Opteron 2352, 16GB RAM): > >>> ciss0: port 0x4000-0x40ff mem >>> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 > on pci80 >>> ciss0: [ITHREAD] >>> da0 at ciss0 bus 0 target 0 lun 0 >>> da0: Fixed Direct Access SCSI-5 device >>> da0: 135.168MB/s transfers >>> da0: Command Queueing Enabled >>> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >>> # dbench -t 10 1 2 3 4 >>> blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec > 192.675 MB/sec >>> blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec > 12.407 MB/sec > >>> Any help/ideas would be greatly appreciated. I have run through > all the >>> Insight diagnostics tools and it fails to find anything wrong with > the slow >>> server. > >>> Cheers, >>> Nathan > >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFKgnX+xJBWvpalMpkRAnVpAKCx95MbkyGOenqUwmuSmwnBa/l2awCglwpV L1cWIc+IauiDJSPn8fsqJKY= =8QiN -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Wed Aug 12 04:53:26 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 741DD106566C for ; Wed, 12 Aug 2009 04:53:26 +0000 (UTC) (envelope-from grarpamp@gmail.com) Received: from mail-ew0-f206.google.com (mail-ew0-f206.google.com [209.85.219.206]) by mx1.freebsd.org (Postfix) with ESMTP id 0B6EB8FC48 for ; Wed, 12 Aug 2009 04:53:25 +0000 (UTC) Received: by ewy2 with SMTP id 2so4194464ewy.43 for ; Tue, 11 Aug 2009 21:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=w9PjcqxNaJeFBWxBp7ZVyqQrICEGPgsWG52md9PTT6E=; b=ryqtmAeZ0mZpbZH3bw08eJtGFGTui0HNLD+AtgoUvC50HXB5cFd34IGCIbHDxUTD54 Ohl809CRM7YAUmIJ1qbBPHHXFZhxQ5jejkLeQFq+dFftpEjx2hzWFDEMF6IxPQ3dZioW CxK3gGi3+2swdg3IDvZuh3a8HtRq9uwDMl+HA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=QaQyg4QFJAlE7UiXR6QtwaTsN/lXbnNEhvIhdyiMi5OE1gPdBi4lYQvY3ffFpQdAGS P7RUjPF1MCE8B/b8nK9LukqnxrY9YmL37XbheYnAFi/vSlJ6fAIZniV/1wn2mjV2sNvy UdAoseXw7nTZ6DQClomOpzFuHQHHHqywo5X5E= MIME-Version: 1.0 Received: by 10.210.70.8 with SMTP id s8mr1874584eba.69.1250052804672; Tue, 11 Aug 2009 21:53:24 -0700 (PDT) Date: Wed, 12 Aug 2009 00:53:23 -0400 Message-ID: From: grarpamp To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 12 Aug 2009 11:13:31 +0000 Subject: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 04:53:26 -0000 > cards aren't going to help with zfs. No, but for geli hifn(4) crypto(4)/(9), geli(8) might work if aes-cbc is indeed the mode geli uses. See the source I guess. > Does anyone make a disk controller with crypto built in? Yes. There are trays and cable dongles and things that do aes/des. And some drives are coming out with it in firmware. Does anyone like the cost, closed-source, and trust model of such hardware? > atacontrol I'm partly up against this because some failing drives are negotiating lower speeds for themselves. Never thought of is as a test tool though. > Pata doesn't do error detection on the control info ZFS handles that and informs user about silent corruption. Precisely because of the chained checksums. It's quite addicting. http://www.opensolaris.org/os/community/zfs/ Your little to no overhead for ffs sounds right as always. From owner-freebsd-performance@FreeBSD.ORG Wed Aug 12 12:14:35 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 763A71065678 for ; Wed, 12 Aug 2009 12:14:35 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 42A188FC16 for ; Wed, 12 Aug 2009 12:14:34 +0000 (UTC) Received: from working (pool-72-95-226-5.pitbpa.ftas.verizon.net [72.95.226.5]) (AUTH: LOGIN wmoran, SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 12 Aug 2009 08:04:31 -0400 id 00056419.000000004A82AFCF.0000D036 Date: Wed, 12 Aug 2009 08:04:34 -0400 From: Bill Moran To: grarpamp Message-Id: <20090812080434.addbf0e0.wmoran@collaborativefusion.com> In-Reply-To: References: Organization: Collaborative Fusion Inc. X-Mailer: Sylpheed 2.7.0 (GTK+ 2.16.5; i386-portbld-freebsd7.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: RELENG_7 heavy disk = system crawls X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 12:14:35 -0000 grarpamp wrote: > > > cards aren't going to help with zfs. > > No, but for geli hifn(4) crypto(4)/(9), geli(8) might work > if aes-cbc is indeed the mode geli uses. See the source I guess. Note that only the most expensive cards are faster than a CPU. Unless you've got a very large budget, you'll get more bang for your buck out of adding CPUs to the unit. During crypto ops, the system will have a spare CPU to use to encrypt, and when not doing crypto ops, you have the added benefit of more processing power for apps. Additionally, support for FreeBSD is spotty. And we've seen cards that work in FreeBSD, but do so at far below their claimed throughput. There are apparently some inefficiencies in the drivers. -- Bill Moran Collaborative Fusion Inc. wmoran@collaborativefusion.com Phone: 412-422-3463x4023 **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. **************************************************************** From owner-freebsd-performance@FreeBSD.ORG Thu Aug 13 05:35:31 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6D3E1065670 for ; Thu, 13 Aug 2009 05:35:31 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 47F1D8FC43 for ; Thu, 13 Aug 2009 05:35:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id B6628C5EF; Thu, 13 Aug 2009 08:35:29 +0300 (EEST) Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 63792-08; Thu, 13 Aug 2009 08:35:28 +0300 (EEST) Received: from [192.168.2.188] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id A2D7FBCF5; Thu, 13 Aug 2009 08:35:28 +0300 (EEST) Message-ID: <4A83A61E.6010009@bulinfo.net> Date: Thu, 13 Aug 2009 08:35:26 +0300 From: Krassimir Slavchev User-Agent: Thunderbird 2.0.0.19 (X11/20090225) MIME-Version: 1.0 To: Nathan Le Nevez References: In-Reply-To: X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at mx.bulinfo.net Cc: "freebsd-performance@freebsd.org" Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 05:35:32 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nathan Le Nevez wrote: > I?m fairly certain this is a hardware problem ? swapping the disks from > a known working install on another blade produced the same lousy > performance. Hmm, Check the SAS cables between the controller and the disks tray. Also check the connector's pins. I had such problems on a DL380. Best Regards > > Thanks for your help, time to put a call in with HP although without any > real errors to show them it is going to be a challenge. > > > On 12/08/09 5:57 PM, "Krassimir Slavchev" wrote: > > Is it possible to exchange disks between your blade1 and blade2 servers? > Or to remove disks from one server and connect them to another? > Also compare 'tunefs -p /' outputs > Also compare the read speed of a raw device with e.g. 'dd if=/dev/da0 > of=/dev/null bs=1m count=100' > > Nathan Le Nevez wrote: >> # df -h >> Filesystem Size Used Avail Capacity Mounted on >> /dev/da0s1a 496M 224M 232M 49% / >> devfs 1.0K 1.0K 0B 100% /dev >> /dev/da0s1e 496M 14K 456M 0% /tmp >> /dev/da0s1f 119G 623M 109G 0% /usr >> /dev/da0s1d 4.8G 346K 4.4G 0% /var >> # mount >> /dev/da0s1a on / (ufs, local) >> devfs on /dev (devfs, local) >> /dev/da0s1e on /tmp (ufs, local, soft-updates) >> /dev/da0s1f on /usr (ufs, local, soft-updates) >> /dev/da0s1d on /var (ufs, local, soft-updates) > >> / - Throughput 6.59862 MB/sec 4 procs >> /usr - Throughput 14.487 MB/sec 4 procs > > > >> On 12/08/09 3:41 PM, "Krassimir Slavchev" wrote: > >> Looks okay. >> How your disks are partitioned and from where you are running dbench. >> Look at the -D option. For example I have: >> / without soft updates -> Throughput 72.7276 MB/sec 4 >> procs >> /var with soft updates -> Throughput 286.528 MB/sec 4 procs > >> Are you sure that you are not running dbench on zfs or encrypted >> partition? > >> Nathan Le Nevez wrote: >>> # vmstat -i >>> interrupt total rate >>> irq1: atkbd0 18 0 >>> irq5: ohci0 ohci1+ 1 0 >>> irq19: ciss0 144916 3 >>> irq21: uhci0 22 0 >>> cpu0: timer 80002970 1999 >>> irq256: bce0 17042 0 >>> cpu2: timer 79994902 1999 >>> cpu1: timer 79994975 1999 >>> cpu3: timer 79995009 1999 >>> cpu6: timer 79994957 1999 >>> cpu5: timer 79995046 1999 >>> cpu4: timer 79995041 1999 >>> cpu7: timer 79995057 1999 >>> Total 640129956 16000 > >>> # camcontrol tags da0 >>> (pass0:ciss0:0:0:0): device openings: 254 > >>> Just for clarification, both systems are running amd64. > >>> Thanks, > >>> Nathan > >>> -----Original Message----- >>> From: Krassimir Slavchev [mailto:krassi@bulinfo.net] >>> Sent: Tuesday, 11 August 2009 9:45 PM >>> To: Nathan Le Nevez >>> Cc: freebsd-performance@freebsd.org >>> Subject: Re: Very slow I/O performance on HP BL465c > >>> Hi, > >>> What is the output of 'vmstat -i' and 'camcontrol tags da0' ? >>> I have a ML350 running 7-STABLE with same controller and disks and >>> performance is almost same as your good server. > >>> Nathan Le Nevez wrote: >>>> Hi, > >>>> I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which >> performs very >>>> poorly. Both have the same RAID controller and 2 x 146GB 10k SAS > disks >>>> configured in RAID-1. Both controllers have write-cache enabled. Both >>>> servers are running the same BIOS and firmware versions. Neither >> servers are >>>> running any services other than sshd. > >>>> Blade with good performance (2 x Opteron 2218, 8GB RAM): > >>>> ciss0: port 0x4000-0x40ff mem >>>> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 >> on pci80 >>>> ciss0: [ITHREAD] >>>> da0 at ciss0 bus 0 target 0 lun 0 >>>> da0: AFPi xCePdU D#i2r Leacuntc >> hAcecde!s >>>> s SCSI-5 device >>>> da0: 135.168MB/s transfers >>>> dSaM0P:: CAoP CPU #3 Launched! >>>> mmand Queueing Enabled >>>> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >>>> Blade with bad performance (2 x Opteron 2352, 16GB RAM): > >>>> ciss0: port 0x4000-0x40ff mem >>>> 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 >> on pci80 >>>> ciss0: [ITHREAD] >>>> da0 at ciss0 bus 0 target 0 lun 0 >>>> da0: Fixed Direct Access SCSI-5 device >>>> da0: 135.168MB/s transfers >>>> da0: Command Queueing Enabled >>>> da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > >>>> # dbench -t 10 1 2 3 4 >>>> blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec >> 192.675 MB/sec >>>> blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec >> 12.407 MB/sec > >>>> Any help/ideas would be greatly appreciated. I have run through >> all the >>>> Insight diagnostics tools and it fails to find anything wrong with >> the slow >>>> server. > >>>> Cheers, >>>> Nathan > >>>> _______________________________________________ >>>> freebsd-performance@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>> To unsubscribe, send any mail to >> "freebsd-performance-unsubscribe@freebsd.org" > > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFKg6YexJBWvpalMpkRAt+EAKCIQFP0lAv44TpcUPm2ZC3rP4opSwCfcOZN Gp4q2KxquOyvYJNYvTPk+Sc= =GRjk -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Thu Aug 13 18:05:56 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1552106566C for ; Thu, 13 Aug 2009 18:05:56 +0000 (UTC) (envelope-from lambert@lambertfam.org) Received: from sysmon.tcworks.net (sysmon.tcworks.net [65.66.76.4]) by mx1.freebsd.org (Postfix) with ESMTP id AD7748FC41 for ; Thu, 13 Aug 2009 18:05:56 +0000 (UTC) Received: from sysmon.tcworks.net (localhost [127.0.0.1]) by sysmon.tcworks.net (8.13.1/8.13.1) with ESMTP id n7DHbRd2064325 for ; Thu, 13 Aug 2009 12:37:27 -0500 (CDT) (envelope-from lambert@lambertfam.org) Received: (from lambert@localhost) by sysmon.tcworks.net (8.13.1/8.13.1/Submit) id n7DHbRph064324 for freebsd-performance@freebsd.org; Thu, 13 Aug 2009 12:37:27 -0500 (CDT) (envelope-from lambert@lambertfam.org) X-Authentication-Warning: sysmon.tcworks.net: lambert set sender to lambert@lambertfam.org using -f Date: Thu, 13 Aug 2009 12:37:27 -0500 From: Scott Lambert To: FreeBSD Performance Message-ID: <20090813173727.GF91291@sysmon.tcworks.net> Mail-Followup-To: FreeBSD Performance References: <4A83A61E.6010009@bulinfo.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4A83A61E.6010009@bulinfo.net> User-Agent: Mutt/1.4.2.2i Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 18:05:57 -0000 > Nathan Le Nevez wrote: > > I?m fairly certain this is a hardware problem ? swapping the disks from > > a known working install on another blade produced the same lousy > > performance. > > >>>> Hi, > >>>> > >>>> I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which > >>>> performs very poorly. Both have the same RAID controller and 2 x > >>>> 146GB 10k SAS disks configured in RAID-1. Both controllers have > >>>> write-cache enabled. Both servers are running the same BIOS and > >>>> firmware versions. Neither servers are running any services other > >>>> than sshd. > >>>> > >>>> Blade with good performance (2 x Opteron 2218, 8GB RAM): > >>>> > >>>> Blade with bad performance (2 x Opteron 2352, 16GB RAM): > >>>> > >>>> # dbench -t 10 1 2 3 4 > >>>> blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/sec > >>>> blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/sec Sorry, I deleted the start of this thread. I figured someone else would suggest pulling half the RAM in the slow server. That seems to be the biggest difference in configuration, other than CPU model. I don't know that it could cause this problem, but it would seem to be easy enough to test if they are not in production, yet. Just grasping at staws here... -- Scott Lambert KC5MLE Unix SysAdmin lambert@lambertfam.org From owner-freebsd-performance@FreeBSD.ORG Thu Aug 13 21:49:25 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B214106568C for ; Thu, 13 Aug 2009 21:49:25 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from services.ipt.ru (services.ipt.ru [194.62.233.110]) by mx1.freebsd.org (Postfix) with ESMTP id 28BE48FC55 for ; Thu, 13 Aug 2009 21:49:24 +0000 (UTC) Received: from [85.173.16.247] (helo=moosi) by services.ipt.ru with esmtpa (Exim 4.54 (FreeBSD)) id 1MbhvI-000EHA-GP; Fri, 14 Aug 2009 01:33:24 +0400 To: Nathan Le Nevez References: From: Boris Samorodov Date: Fri, 14 Aug 2009 01:34:45 +0400 In-Reply-To: (Nathan Le Nevez's message of "Tue\, 11 Aug 2009 09\:53\:47 +1000") Message-ID: <10994730@ipt.ru> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "freebsd-performance@freebsd.org" Subject: Re: Very slow I/O performance on HP BL465c X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 21:49:25 -0000 Nathan Le Nevez writes: > I'm running 7.2-p3 on 2x HP BL465c blade servers, one of which performs very > poorly. Both have the same RAID controller and 2 x 146GB 10k SAS disks > configured in RAID-1. Both controllers have write-cache enabled. Both > servers are running the same BIOS and firmware versions. Neither servers are > running any services other than sshd. > > Blade with good performance (2 x Opteron 2218, 8GB RAM): > > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: AFPi xCePdU D#i2r Leacuntc hAcecde!s > s SCSI-5 device > da0: 135.168MB/s transfers > dSaM0P:: CAoP CPU #3 Launched! > mmand Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > > Blade with bad performance (2 x Opteron 2352, 16GB RAM): > > ciss0: port 0x4000-0x40ff mem > 0xfdf80000-0xfdffffff,0xfdf70000-0xfdf77fff irq 19 at device 8.0 on pci80 > ciss0: [ITHREAD] > da0 at ciss0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 135.168MB/s transfers > da0: Command Queueing Enabled > da0: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) > > # dbench -t 10 1 2 3 4 > blade1 183.456 MB/sec 236.86 MB/sec 299.28 MB/sec 192.675 MB/sec > blade2 6.97931 MB/sec 9.42293 MB/sec 10.2482 MB/sec 12.407 MB/sec > > Any help/ideas would be greatly appreciated. I have run through all the > Insight diagnostics tools and it fails to find anything wrong with the slow > server. The description of those blades at HP site says "HP Smart Array E200i storage controller with 64MB read cache and optional battery-backed write cache". Does both controllers have batteries? I've seen very poor perfomance at RAIDs (with write cache enabled) without batteries. -- WBR, bsam From owner-freebsd-performance@FreeBSD.ORG Fri Aug 14 21:45:25 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C30E4106568D for ; Fri, 14 Aug 2009 21:45:25 +0000 (UTC) (envelope-from cpardo@fastsoft.com) Received: from HQ-ES.FASTSOFT.COM (hq-es.fastsoft.com [38.102.243.86]) by mx1.freebsd.org (Postfix) with ESMTP id A43268FC61 for ; Fri, 14 Aug 2009 21:45:25 +0000 (UTC) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Date: Fri, 14 Aug 2009 14:33:25 -0700 Message-ID: In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Test on 10GBE Intel based network card Thread-Index: AcoVIm0LnEUSix6PQmaZQulzWV0J9QAksHgAAdwK7iA= References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local><0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com><36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local><4A77094C.8030308@elischer.org><36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local><4A785F20.8050807@elischer.org><2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> From: "Carlos Pardo" To: "Jack Vogel" Cc: freebsd-performance@freebsd.org Subject: RE: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2009 21:45:25 -0000 Hi Jack, I have a quick question. We are getting too many missed packets per = minute running about 3Gbs traffic. We can not use frame control in our = application. We are assuming that there is no way to improve upon the = problem since it seems to be a hardware limitation with the receive = FIFO. We are using the Intel=AE 82598EB 10 Gigabit Ethernet Controller. = When can we expect the next generation card from Intel? Thanks for any = information you may provide. =20 Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. Best Regards, Cpardo -----Original Message----- From: owner-freebsd-performance@freebsd.org = [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi = Fabrizio Sent: Wednesday, August 05, 2009 3:13 AM To: Jack Vogel; Julian Elischer Cc: freebsd-performance@freebsd.org; Stefan Lambrev Subject: RE: Test on 10GBE Intel based network card No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver = :<((((( ++fabrizio ------------------------------------------------------------------ Telecom Italia Fabrizio INVERNIZZI Technology - TILAB Accesso Fisso e Trasporto Via Reiss Romoli, 274 10148 Torino Tel. +39 011 2285497 Mob. +39 3316001344 Fax +39 06 41867287 ________________________________ From: Jack Vogel [mailto:jfvogel@gmail.com] Sent: marted=EC 4 agosto 2009 18.42 To: Julian Elischer Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev Subject: Re: Test on 10GBE Intel based network card Your nmbclusters is very low, you list it twice so I'm assuming the = second value is what it ends up being, 32K :( I would set it to: kern.ipc.nmbclusters=3D262144 Also, I thought you were using the current driver, but now it looks like = you are using something fairly old, use my latest code which is 1.8.8 Jack On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer = > wrote: Invernizzi Fabrizio wrote: The limitation that you see is about the max number of packets that FreeBSD can handle - it looks like your best performance is reached at 64 byte packets? If you are meaning in term of Packet per second, you are right. These are the packet per second measured during tests: 64 byte: 610119 Pps 512 byte: 516917 Pps 1492 byte: 464962 Pps Am I correct that the maximum you can reach is around 639,000 packets per second? Yes, as you can see the maximum is 610119 Pps. Where does this limit come from? ah that's the whole point of tuning :-) there are severalpossibities: 1/ the card's interrupts are probably attache dto aonly 1 cpu, so that cpu can do no more work This seems not to be the problem. See below a top snapshot during a = 64byte-long packet storm last pid: 8552; load averages: 0.40, 0.09, 0.03 = up = 0+20:36:58 09:40:29 124 processes: 13 running, 73 sleeping, 38 waiting CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free Swap: 2048M Total, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: = cpu3 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: = cpu0 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: = cpu2 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: = cpu1 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 = rxq 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 = rxq 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 = rxq 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 = rxq .... It looks like the 4 rxq processes are bound to the 4 available cores = with equal distribution. 2/ if more than 1 cpu is working, it may be that there is a lock in heavy contention somewhere. This I think is the problem. I am trying to understand how to 1- see where the heavy contention is (context switching? Some limiting = setting?) 2- mitigate it there ia a lock profiling tool that right now I can't remember the name = of.. look it up with google :-) FreeBSD lock profiling tool ah, first hit... http://blogs.epfl.ch/article/23832 is the machine still responsive to other networks while running at maximum capacity on this network? (make sure that the other networks are on a differnet CPU (hmm I can't remember how to do that :-). Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. _______________________________________________ freebsd-performance@freebsd.org = mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle = persone indicate. La diffusione, copia o qualsiasi altra azione = derivante dalla conoscenza di queste informazioni sono rigorosamente = vietate. Qualora abbiate ricevuto questo documento per errore siete = cortesemente pregati di darne immediata comunicazione al mittente e di = provvedere alla sua distruzione, Grazie. This e-mail and any attachments is confidential and may contain = privileged information intended for the addressee(s) only. = Dissemination, copying, printing or use by anybody else is unauthorised. = If you are not the intended recipient, please delete this message and = any attachments and advise the sender by return e-mail, Thanks. [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. = Non stampare questa mail se non =E8 necessario. From owner-freebsd-performance@FreeBSD.ORG Fri Aug 14 22:15:30 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7726106568C for ; Fri, 14 Aug 2009 22:15:30 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-yw0-f199.google.com (mail-yw0-f199.google.com [209.85.211.199]) by mx1.freebsd.org (Postfix) with ESMTP id 1C7E18FC61 for ; Fri, 14 Aug 2009 22:15:29 +0000 (UTC) Received: by ywh37 with SMTP id 37so2647130ywh.28 for ; Fri, 14 Aug 2009 15:15:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=rPcxQCDy1wYA7ChG/LFrNBvmNRAlck6eK7nQzs3JiaY=; b=ZHb92VapXs9cRSUqZ+zdNYZXzZAfC7bOYNe2y7mHa7i8RGC5vjW7JRV0FbD8LCnfcW V63Gz2qEX8dfTL9PrN9dUnEgKQwlC/yoWsLTxeVaDn8mWQs9PzUPcoNgq4Nw3MS1KntH xS65HqFxLjxyqA8C8SG0rPlAVwGi2SeqBa8fY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OoCZGztgbZbtA4dOQDMmlrI7CxedWyG8fscQjgHklUPgJDTTNr4O9xBIeM4qauK3Go ege5qxBoHiM6lzBDlklvNno2zKgN2Kutd+M0k1k6UkBzF5G19ho8kHABqKPXhe8k9Esi OsWFxKIxIaCJRLD2/RvW/necimLGANtQSpWcw= MIME-Version: 1.0 Received: by 10.100.107.3 with SMTP id f3mr2143056anc.87.1250288129258; Fri, 14 Aug 2009 15:15:29 -0700 (PDT) In-Reply-To: References: <36A93B31228D3B49B691AD31652BCAE9A4560DF911@GRFMBX702BA020.griffon.local> <0E567C7E-4EAA-4B89-9A8D-FD0450D32ED7@moneybookers.com> <36A93B31228D3B49B691AD31652BCAE9A4560DF947@GRFMBX702BA020.griffon.local> <4A77094C.8030308@elischer.org> <36A93B31228D3B49B691AD31652BCAE9A45696721F@GRFMBX702BA020.griffon.local> <4A785F20.8050807@elischer.org> <2a41acea0908040941y39f16c8cocb84b001e1e9f0de@mail.gmail.com> <36A93B31228D3B49B691AD31652BCAE9A45696743F@GRFMBX702BA020.griffon.local> Date: Fri, 14 Aug 2009 15:15:29 -0700 Message-ID: <2a41acea0908141515o45b7c74g40dbddc32d4b754b@mail.gmail.com> From: Jack Vogel To: Carlos Pardo Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Test on 10GBE Intel based network card X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2009 22:15:30 -0000 I've talked over the issues with the guy on our team who has been most involved in 10G performance, he asserts that 3Gbs will saturate a single cpu with a small packet size, this is why you need multiqueue across multiple cores. He was dubious about the FIFO assertion, its a relative thing, if you can keep the thing drained it won't be a problem, doing that is a complex combination of factors, the cpu, the bus, the memory.... If you don't deal with the systemic issues just cuz you go from an 82598 to a 82599 is not going to solve things. What about LRO, are/can you use that? I never saw an answer about the forwarding question, you can't use LRO in that case of course. Regards, Jack On Fri, Aug 14, 2009 at 2:33 PM, Carlos Pardo wrote: > Hi Jack, > > I have a quick question. We are getting too many missed packets per minut= e > running about 3Gbs traffic. We can not use frame control in our applicati= on. > We are assuming that there is no way to improve upon the problem since it > seems to be a hardware limitation with the receive FIFO. We are using the > Intel=AE 82598EB 10 Gigabit Ethernet Controller. When can we expect the n= ext > generation card from Intel? Thanks for any information you may provide. > > Typical error count "ix0: Missed Packets =3D 81174" after a few minutes. > > Best Regards, > > Cpardo > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org [mailto: > owner-freebsd-performance@freebsd.org] On Behalf Of Invernizzi Fabrizio > Sent: Wednesday, August 05, 2009 3:13 AM > To: Jack Vogel; Julian Elischer > Cc: freebsd-performance@freebsd.org; Stefan Lambrev > Subject: RE: Test on 10GBE Intel based network card > > No improvement with kern.ipc.nmbclusters=3D262144 and 1.8.6 driver :<((((= ( > > ++fabrizio > > ------------------------------------------------------------------ > Telecom Italia > Fabrizio INVERNIZZI > Technology - TILAB > Accesso Fisso e Trasporto > Via Reiss Romoli, 274 10148 Torino > Tel. +39 011 2285497 > Mob. +39 3316001344 > Fax +39 06 41867287 > > > ________________________________ > From: Jack Vogel [mailto:jfvogel@gmail.com] > Sent: marted=EC 4 agosto 2009 18.42 > To: Julian Elischer > Cc: Invernizzi Fabrizio; freebsd-performance@freebsd.org; Stefan Lambrev > Subject: Re: Test on 10GBE Intel based network card > > Your nmbclusters is very low, you list it twice so I'm assuming the secon= d > value is > what it ends up being, 32K :( > > I would set it to: > > kern.ipc.nmbclusters=3D262144 > > Also, I thought you were using the current driver, but now it looks like > you are > using something fairly old, use my latest code which is 1.8.8 > > Jack > > > On Tue, Aug 4, 2009 at 9:17 AM, Julian Elischer > wrote: > Invernizzi Fabrizio wrote: > The limitation that you see is about the max number of packets that > FreeBSD can handle - it looks like your best performance is reached at > 64 byte packets? > If you are meaning in term of Packet per second, you are right. These > are the packet per second measured during tests: > 64 byte: 610119 Pps > 512 byte: 516917 Pps > 1492 byte: 464962 Pps > > > Am I correct that the maximum you can reach is around 639,000 packets > per second? > Yes, as you can see the maximum is 610119 Pps. > Where does this limit come from? > ah that's the whole point of tuning :-) > there are severalpossibities: > 1/ the card's interrupts are probably attache dto aonly 1 cpu, > so that cpu can do no more work > > This seems not to be the problem. See below a top snapshot during a > 64byte-long packet storm > > last pid: 8552; load averages: 0.40, 0.09, 0.03 > up > 0+20:36:58 09:40:29 > 124 processes: 13 running, 73 sleeping, 38 waiting > CPU: 0.0% user, 0.0% nice, 86.3% system, 12.3% interrupt, 1.5% idle > Mem: 13M Active, 329M Inact, 372M Wired, 68K Cache, 399M Buf, 7207M Free > Swap: 2048M Total, 2048M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAN= D > 11 root 1 171 ki31 0K 16K RUN 3 20.2H 51.17% idle: > cpu3 > 14 root 1 171 ki31 0K 16K RUN 0 20.2H 50.88% idle: > cpu0 > 12 root 1 171 ki31 0K 16K RUN 2 20.2H 50.49% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 20.2H 50.10% idle: > cpu1 > 42 root 1 -68 - 0K 16K RUN 1 14:20 36.47% ix0 rxq > 38 root 1 -68 - 0K 16K CPU0 0 14:15 36.08% ix0 rxq > 44 root 1 -68 - 0K 16K CPU2 2 14:08 34.47% ix0 rxq > 40 root 1 -68 - 0K 16K CPU3 3 13:42 32.37% ix0 rxq > .... > > It looks like the 4 rxq processes are bound to the 4 available cores with > equal distribution. > > > > 2/ if more than 1 cpu is working, it may be that there is a lock in > heavy contention somewhere. > > This I think is the problem. I am trying to understand how to > 1- see where the heavy contention is (context switching? Some limiting > setting?) > 2- mitigate it > > > > there ia a lock profiling tool that right now I can't remember the name > of.. > > look it up with google :-) FreeBSD lock profiling tool > > ah, first hit... > > http://blogs.epfl.ch/article/23832 > > > > is the machine still responsive to other networks while > running at maximum capacity on this network? (make sure that > the other networks are on a differnet CPU (hmm I can't remember how to > do that :-). > > > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > _______________________________________________ > freebsd-performance@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org freebsd-performance-unsubscribe@freebsd.org>" > > Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle > persone indicate. La diffusione, copia o qualsiasi altra azione derivante > dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualo= ra > abbiate ricevuto questo documento per errore siete cortesemente pregati d= i > darne immediata comunicazione al mittente e di provvedere alla sua > distruzione, Grazie. > > This e-mail and any attachments is confidential and may contain privilege= d > information intended for the addressee(s) only. Dissemination, copying, > printing or use by anybody else is unauthorised. If you are not the inten= ded > recipient, please delete this message and any attachments and advise the > sender by return e-mail, Thanks. > > [cid:00000000000000000000000000000001@TI.Disclaimer]Rispetta l'ambiente. > Non stampare questa mail se non =E8 necessario. > >