From owner-freebsd-performance@FreeBSD.ORG Wed Sep 9 08:58:02 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC91C106566C; Wed, 9 Sep 2009 08:58:02 +0000 (UTC) (envelope-from tlott@gamesnet.de) Received: from spirit.gamesnet.de (cl-858.dus-01.de.sixxs.net [IPv6:2a01:198:200:359::2]) by mx1.freebsd.org (Postfix) with ESMTP id 7D0FC8FC13; Wed, 9 Sep 2009 08:58:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by spirit.gamesnet.de (Postfix) with ESMTP id D00203047A6; Wed, 9 Sep 2009 10:58:01 +0200 (CEST) X-Virus-Scanned: amavisd-new at gamesnet.de Received: from spirit.gamesnet.de ([127.0.0.1]) by localhost (spirit.gamesnet.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zd9st0t3LU4F; Wed, 9 Sep 2009 10:57:58 +0200 (CEST) Received: from sub.han.vpn.gamesnet.de (sub.han.vpn.gamesnet.de [192.168.1.101]) by spirit.gamesnet.de (Postfix) with ESMTPSA id 31217304736; Wed, 9 Sep 2009 10:48:08 +0200 (CEST) Date: Wed, 9 Sep 2009 10:48:07 +0200 From: Tobias Lott To: freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20090909104807.633418ea@sub.han.vpn.gamesnet.de> In-Reply-To: <20090908121945.43ba3976@sub.han.vpn.gamesnet.de> References: <20090908121945.43ba3976@sub.han.vpn.gamesnet.de> X-Mailer: Claws Mail 3.7.2 (GTK+ 2.16.6; i386-portbld-freebsd7.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Subject: Re: System crawls after Upgrade 7.0->7.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2009 08:58:02 -0000 On Tue, 8 Sep 2009 12:19:45 +0200 Tobias Lott wrote: > Hey Everyone, > > I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS > related Stuff is located on an UFS Slice, Application is on a ZFS > Volume. > > After the Upgrade everything seemed fine, but a User noticed one PHP > Script which is basically loading a plain Textfile into Mysql times > out. PHP Timeout was set to 60 secs, that was more then enough just > one day before the upgrade. > > System-wise its like Mysql can't get get the data fast enough, process > is max at 10% cpu usage most of the time in sbwait state. > > Haven't changed any sysctl (kern.maxvnodes="200000", > vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="100M" > but had those before already) or kernel parameter at that time. But > according to the Wiki's ZFS Tuning Guide those aren't needed anymore > so I tried without but no change there either. > > I checked the following (and the current used values each): > maxproc > maxfiles > kern.ipc.somaxconn > kern.ipc.nmbclusters > > Getting no error messages dmesg, syslog wise. > > Tried moving Mysql to the UFS Slice but same thing happens happens > there. > > Some very basic testing like dd-ing: > # dd if=/dev/da0 of=/dev/null bs=1024 count=1048576 > 1048576+0 records in > 1048576+0 records out > 1073741824 bytes transferred in 157.698602 secs (6808823 bytes/sec) > > FreeBSD hostname 7.2-STABLE FreeBSD 7.2-STABLE #3 r196954: > Tue Sep 8 02:10:22 CEST 2009 > root@hostname:/usr/obj/usr/src/sys/SPIRIT amd64 > > Since I upgraded the Machines zpool to version 13 already I don't > really wanna go back to 7.0, but it seems the only way out atm. > > Hopefully someone can give me a Hint where maybe I forgot to check. > > > Somehow it feels like the last Versions, I'd say starting with 7.x > doesn't really feel that Rock-Stable as Versions Prior used to > be. Prolly cause I'm using an Experimental Feature like ZFS, so no > Offense nor Blame and this is not intended as a Flame or whatsoever! > So don't get me wrong, I always liked, used, recommended FreeBSD since > 4.0 and will continue to do so! > > Best Regards > I've tested some more it seems the Bottleneck is the HD, Raid Array is Optimal checked it. Running bonnie++ takes like forever, couldn't finish it since its a Productive Machine, cause it renders the Server useless, every try to access the server while bonnie is running times out. Weird thing is bonnie++ is using max 1-3% CPU. But according to gstat its only ~3mb/s (displayed as 100% busy) during first 3 Tests that is. savecore: reboot after panic: page fault But had no luck with the coredumb # kgdb kernel.debug /var/crash/vmcore.12 Cannot access memory at address 0x0 Or is there any other method I could try? Cause this is getting nasty unstable -- Tobias Lott From owner-freebsd-performance@FreeBSD.ORG Wed Sep 9 17:02:15 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1221106568F; Wed, 9 Sep 2009 17:02:15 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: from outgoing01.lava.net (cake.lava.net [IPv6:2001:1888:0:1:230:48ff:fe5b:3b50]) by mx1.freebsd.org (Postfix) with ESMTP id 88A768FC1C; Wed, 9 Sep 2009 17:02:15 +0000 (UTC) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by outgoing01.lava.net (Postfix) with ESMTP id C2C84D0510; Wed, 9 Sep 2009 07:02:14 -1000 (HST) Received: by malasada.lava.net (Postfix, from userid 102) id 0942F153882; Wed, 9 Sep 2009 07:02:13 -1000 (HST) Date: Wed, 9 Sep 2009 07:02:13 -1000 From: Clifton Royston To: Tobias Lott Message-ID: <20090909170213.GA7925@lava.net> Mail-Followup-To: Tobias Lott , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org References: <20090908121945.43ba3976@sub.han.vpn.gamesnet.de> <20090909104807.633418ea@sub.han.vpn.gamesnet.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090909104807.633418ea@sub.han.vpn.gamesnet.de> User-Agent: Mutt/1.4.2.2i X-Mailman-Approved-At: Wed, 09 Sep 2009 17:38:27 +0000 Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: System crawls after Upgrade 7.0->7.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2009 17:02:15 -0000 On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote: > On Tue, 8 Sep 2009 12:19:45 +0200 > Tobias Lott wrote: > > Hey Everyone, > > > > I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS > > related Stuff is located on an UFS Slice, Application is on a ZFS > > Volume. > > > > After the Upgrade everything seemed fine, but a User noticed one PHP > > Script which is basically loading a plain Textfile into Mysql times > > out. PHP Timeout was set to 60 secs, that was more then enough just > > one day before the upgrade. ... > > Since I upgraded the Machines zpool to version 13 already I don't > > really wanna go back to 7.0, but it seems the only way out atm. > > > > Hopefully someone can give me a Hint where maybe I forgot to check. > > > > Somehow it feels like the last Versions, I'd say starting with 7.x > > doesn't really feel that Rock-Stable as Versions Prior used to > > be. Prolly cause I'm using an Experimental Feature like ZFS, so no > > Offense nor Blame and this is not intended as a Flame or whatsoever! > > So don't get me wrong, I always liked, used, recommended FreeBSD since > > 4.0 and will continue to do so! > > > > Best Regards > > > > I've tested some more it seems the Bottleneck is the HD, Raid Array is > Optimal checked it. Probably you already checked this, but did you go through the dmesg output relating to ata and drive detection closely? In the past from time to time I've had machines suddenly start crawling after an upgrade, and it turned out it was because some change in the driver detection caused the ata to fall back to pio mode. The symptoms are pretty much like you describe - all HD IO takes forever. -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services From owner-freebsd-performance@FreeBSD.ORG Wed Sep 9 20:00:11 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 489111065694; Wed, 9 Sep 2009 20:00:11 +0000 (UTC) (envelope-from tlott@gamesnet.de) Received: from spirit.gamesnet.de (cl-858.dus-01.de.sixxs.net [IPv6:2a01:198:200:359::2]) by mx1.freebsd.org (Postfix) with ESMTP id 0FF628FC12; Wed, 9 Sep 2009 20:00:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by spirit.gamesnet.de (Postfix) with ESMTP id EA1E0304469; Wed, 9 Sep 2009 22:00:07 +0200 (CEST) X-Virus-Scanned: amavisd-new at gamesnet.de Received: from spirit.gamesnet.de ([127.0.0.1]) by localhost (spirit.gamesnet.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g99aQ5CYov6x; Wed, 9 Sep 2009 22:00:01 +0200 (CEST) Received: from sub.han.vpn.gamesnet.de (sub.han.vpn.gamesnet.de [192.168.1.101]) by spirit.gamesnet.de (Postfix) with ESMTPSA id 3765A304457; Wed, 9 Sep 2009 22:00:01 +0200 (CEST) Date: Wed, 9 Sep 2009 21:59:59 +0200 From: Tobias Lott To: freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20090909215959.6b5152f2@sub.han.vpn.gamesnet.de> In-Reply-To: <20090909213022.34d64e47@sub.han.vpn.gamesnet.de> References: <20090908121945.43ba3976@sub.han.vpn.gamesnet.de> <20090909104807.633418ea@sub.han.vpn.gamesnet.de> <20090909170213.GA7925@lava.net> <20090909213022.34d64e47@sub.han.vpn.gamesnet.de> X-Mailer: Claws Mail 3.7.2 (GTK+ 2.16.6; i386-portbld-freebsd7.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Subject: Re: System crawls after Upgrade 7.0->7.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2009 20:00:11 -0000 On Wed, 9 Sep 2009 21:30:22 +0200 Tobias Lott wrote: > > > On Wed, 9 Sep 2009 07:02:13 -1000 > Clifton Royston wrote: > > > On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote: > > > On Tue, 8 Sep 2009 12:19:45 +0200 > > > Tobias Lott wrote: > > > > Hey Everyone, > > > > > > > > I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all > > > > OS related Stuff is located on an UFS Slice, Application is on > > > > a ZFS Volume. > > > > > > > > After the Upgrade everything seemed fine, but a User noticed one > > > > PHP Script which is basically loading a plain Textfile into > > > > Mysql times out. PHP Timeout was set to 60 secs, that was more > > > > then enough just one day before the upgrade. > > ... > > > > Since I upgraded the Machines zpool to version 13 already I > > > > don't really wanna go back to 7.0, but it seems the only way > > > > out atm. > > > > > > > > Hopefully someone can give me a Hint where maybe I forgot to > > > > check. > > > > > > > > Somehow it feels like the last Versions, I'd say starting with > > > > 7.x doesn't really feel that Rock-Stable as Versions Prior used > > > > to be. Prolly cause I'm using an Experimental Feature like ZFS, > > > > so no Offense nor Blame and this is not intended as a Flame or > > > > whatsoever! So don't get me wrong, I always liked, used, > > > > recommended FreeBSD since 4.0 and will continue to do so! > > > > > > > > Best Regards > > > > > > > > > > I've tested some more it seems the Bottleneck is the HD, Raid > > > Array is Optimal checked it. > > > > Probably you already checked this, but did you go through the > > dmesg output relating to ata and drive detection closely? > > > > In the past from time to time I've had machines suddenly start > > crawling after an upgrade, and it turned out it was because some > > change in the driver detection caused the ata to fall back to pio > > mode. The symptoms are pretty much like you describe - all HD IO > > takes forever. > > > > -- Clifton > > > > Thanks for that Hint, checked again to be sure, but thats not the > Case. > > mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority-ReSync ) > mpt0:vol0(mpt0:0:0): Using Spare Pool: 0 > mpt0:vol0(mpt0:0:0): 2 Members: > (mpt0:1:32:0): Primary Online > (mpt0:1:1:0): Secondary Online > mpt0:vol0(mpt0:0:0): RAID-1 - Optimal > mpt0:vol0(mpt0:0:0): Status ( Enabled ) > (mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0) > (mpt0:vol0:1): Online > (mpt0:vol0:0): Physical (mpt0:0:32:0), Pass-thru (mpt0:1:1:0) > (mpt0:vol0:0): Online > acd0: CDROM at ata0-master UDMA33 > acd1: CDROM at ata2-slave PIO3 > da0 at mpt0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing Enabled > SdMaP0:: A1P5 1C6P3U4 M#B1 (L3a1u0n5c4h6e4d3!2 > 512 byte sectors: 255H 63S/T 19330C) > > Oh well just did some more research and found: http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02461.html Gonna try it out later and gonna give a Report. -- Tobias Lott From owner-freebsd-performance@FreeBSD.ORG Fri Sep 11 21:21:39 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D729106568D for ; Fri, 11 Sep 2009 21:21:39 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 049D88FC12 for ; Fri, 11 Sep 2009 21:21:38 +0000 (UTC) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 80CE61B16815; Fri, 11 Sep 2009 23:21:37 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.5 Received: from maylo.moneybookers.com (maylo.dev.moneybookers.net [192.168.3.20]) by blah.sun-fish.com (Postfix) with ESMTP id E061E1B16814; Fri, 11 Sep 2009 23:21:36 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by maylo.moneybookers.com (Postfix) with ESMTP id DB94A37BF843; Fri, 11 Sep 2009 23:21:36 +0200 (CEST) Received: from maylo.moneybookers.com ([127.0.0.1]) by localhost (maylo.dev.moneybookers.net [127.0.0.1]) (amavisd-new, port 10024) with LMTP id qn2RkKCZJI-p; Fri, 11 Sep 2009 23:21:34 +0200 (CEST) Received: from [10.1.1.3] (unknown [192.168.25.6]) by maylo.moneybookers.com (Postfix) with ESMTP id 175AC37BF83E; Fri, 11 Sep 2009 23:21:34 +0200 (CEST) Message-Id: From: Stefan Lambrev To: Tobias Lott In-Reply-To: <20090909215959.6b5152f2@sub.han.vpn.gamesnet.de> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Sat, 12 Sep 2009 00:21:33 +0300 References: <20090908121945.43ba3976@sub.han.vpn.gamesnet.de> <20090909104807.633418ea@sub.han.vpn.gamesnet.de> <20090909170213.GA7925@lava.net> <20090909213022.34d64e47@sub.han.vpn.gamesnet.de> <20090909215959.6b5152f2@sub.han.vpn.gamesnet.de> X-Mailer: Apple Mail (2.936) X-Virus-Scanned: ClamAV 0.94/9797/Fri Sep 11 15:59:58 2009 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: System crawls after Upgrade 7.0->7.2 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2009 21:21:39 -0000 Hi Tobias, This is the "pseudo" RAID controller that comes by default with most Dell, low cost HP servers and probably others? If you depend on I/O I would recommend you to change this raid with PERC6/i (looks like this is Dell server) which is little more expensive, but comes with 256MB battery backed cache. In my experience mpt devices are crap, and their performance is worse then software mirror. PERC6/i is something totally different :) It's again LSI product branded from Dell and it works with mfi driver. The only problem in the past with them was the management utility which was linux binary, but today there is official native tool in ports, and in 8.0 in base install there will be a tool to manage your controller. Well this wasn't that big problem compared to mpt which cannot be managed under FreeBSD at all? On Sep 9, 2009, at 10:59 PM, Tobias Lott wrote: > > > On Wed, 9 Sep 2009 21:30:22 +0200 > Tobias Lott wrote: > >> >> >> On Wed, 9 Sep 2009 07:02:13 -1000 >> Clifton Royston wrote: >> >>> On Wed, Sep 09, 2009 at 10:48:07AM +0200, Tobias Lott wrote: >>>> On Tue, 8 Sep 2009 12:19:45 +0200 >>>> Tobias Lott wrote: >>>>> Hey Everyone, >>>>> >>>>> I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all >>>>> OS related Stuff is located on an UFS Slice, Application is on >>>>> a ZFS Volume. >>>>> >>>>> After the Upgrade everything seemed fine, but a User noticed one >>>>> PHP Script which is basically loading a plain Textfile into >>>>> Mysql times out. PHP Timeout was set to 60 secs, that was more >>>>> then enough just one day before the upgrade. >>> ... >>>>> Since I upgraded the Machines zpool to version 13 already I >>>>> don't really wanna go back to 7.0, but it seems the only way >>>>> out atm. >>>>> >>>>> Hopefully someone can give me a Hint where maybe I forgot to >>>>> check. >>>>> >>>>> Somehow it feels like the last Versions, I'd say starting with >>>>> 7.x doesn't really feel that Rock-Stable as Versions Prior used >>>>> to be. Prolly cause I'm using an Experimental Feature like ZFS, >>>>> so no Offense nor Blame and this is not intended as a Flame or >>>>> whatsoever! So don't get me wrong, I always liked, used, >>>>> recommended FreeBSD since 4.0 and will continue to do so! >>>>> >>>>> Best Regards >>>>> >>>> >>>> I've tested some more it seems the Bottleneck is the HD, Raid >>>> Array is Optimal checked it. >>> >>> Probably you already checked this, but did you go through the >>> dmesg output relating to ata and drive detection closely? >>> >>> In the past from time to time I've had machines suddenly start >>> crawling after an upgrade, and it turned out it was because some >>> change in the driver detection caused the ata to fall back to pio >>> mode. The symptoms are pretty much like you describe - all HD IO >>> takes forever. >>> >>> -- Clifton >>> >> >> Thanks for that Hint, checked again to be sure, but thats not the >> Case. >> >> mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority- >> ReSync ) >> mpt0:vol0(mpt0:0:0): Using Spare Pool: 0 >> mpt0:vol0(mpt0:0:0): 2 Members: >> (mpt0:1:32:0): Primary Online >> (mpt0:1:1:0): Secondary Online >> mpt0:vol0(mpt0:0:0): RAID-1 - Optimal >> mpt0:vol0(mpt0:0:0): Status ( Enabled ) >> (mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:0:0) >> (mpt0:vol0:1): Online >> (mpt0:vol0:0): Physical (mpt0:0:32:0), Pass-thru (mpt0:1:1:0) >> (mpt0:vol0:0): Online >> acd0: CDROM at ata0-master UDMA33 >> acd1: CDROM at ata2-slave PIO3 >> da0 at mpt0 bus 0 target 0 lun 0 >> da0: Fixed Direct Access SCSI-5 device >> da0: 300.000MB/s transfers >> da0: Command Queueing Enabled >> SdMaP0:: A1P5 1C6P3U4 M#B1 (L3a1u0n5c4h6e4d3!2 >> 512 byte sectors: 255H 63S/T 19330C) >> >> > > Oh well just did some more research and found: > http://www.mail-archive.com/freebsd-performance@freebsd.org/msg02461.html > > Gonna try it out later and gonna give a Report. > > -- > Tobias Lott > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > " -- Best Wishes, Stefan Lambrev ICQ# 24134177