From owner-freebsd-performance@FreeBSD.ORG Sun Jan 27 11:50:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2471816A417 for ; Sun, 27 Jan 2008 11:50:50 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from spork.qfe3.net (spork.qfe3.net [212.13.207.101]) by mx1.freebsd.org (Postfix) with ESMTP id DB36213C455 for ; Sun, 27 Jan 2008 11:50:49 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from [81.104.123.28] (helo=voi.aagh.net) by spork.qfe3.net with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1JJ5hH-0002cB-SH; Sun, 27 Jan 2008 11:29:11 +0000 Received: from freaky by voi.aagh.net with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JJ5hH-0004XD-Ha; Sun, 27 Jan 2008 11:29:11 +0000 Date: Sun, 27 Jan 2008 11:29:11 +0000 From: Thomas Hurst To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <20080127112911.GA4024@voi.aagh.net> Mail-Followup-To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <479B1185.8020604@quip.cz> Organization: Not much. User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Thomas Hurst Cc: freebsd-performance@freebsd.org Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 11:50:50 -0000 * Miroslav Lachman (000.fbsd@quip.cz) wrote: > I found a painful performance problem with Apache + PHP 5 when > open_basedir directive is enabled. Looks like it's lstat()/readlink() overhead. I wrote a simple bit of PHP similar to yours, but doing 10 * 1000 require "foo/%d/%d.php" calls =66rom the command line: -% time ktrace php main.php Real: 0:09.61 CPU: 99.8% (3.725/5.885) Page: 0 Swap: 0 I/O: (0/0) Mem: 13704 -% time ktrace php -d open_basedir=3D'/home/freaky/openbasedir/foo' main.php Real: 0:16.21 CPU: 86.4% (8.185/5.840) Page: 0 Swap: 0 I/O: (0/0) Mem: 13696 Without open_basedir, a simple script to parse the kdump syscall times produces: lstat : 1.147s/70065 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s fcntl : 0.408s/60007 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s sigprocmask : 0.229s/40311 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s open : 0.223s/10085 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s With open_basedir: lstat : 4.182s/270065 calls =3D 0.000s per call. Max=3D0.005s Mi= n=3D0.0000s readlink : 2.142s/10006 calls =3D 0.000s per call. Max=3D2.020s Min= =3D0.0000s fcntl : 0.421s/60007 calls =3D 0.000s per call. Max=3D0.002s Min= =3D0.0000s close : 0.295s/10085 calls =3D 0.000s per call. Max=3D0.115s Min= =3D0.0000s sigprocmask : 0.237s/40311 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s open : 0.222s/10085 calls =3D 0.000s per call. Max=3D0.000s Min= =3D0.0000s The top two syscalls seem to account for most of the 6.mumble seconds of additional runtime; presumably these are much cheaper on Linux. --=20 Thomas 'Freaky' Hurst http://hur.st/ From owner-freebsd-performance@FreeBSD.ORG Sun Jan 27 12:25:57 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ADEBF16A417 for ; Sun, 27 Jan 2008 12:25:57 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from mail.sweeplist.com (mail.sweeplist.com [77.93.199.51]) by mx1.freebsd.org (Postfix) with ESMTP id 6F18813C447 for ; Sun, 27 Jan 2008 12:25:57 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from [192.168.1.2] (r3a200.net.upc.cz [213.220.192.200]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sweeplist.com (Postfix) with ESMTPSA id 0F2EB67808; Sun, 27 Jan 2008 13:25:56 +0100 (CET) Message-ID: <479C7859.2060703@quip.cz> Date: Sun, 27 Jan 2008 13:26:01 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: Thomas Hurst References: <479B1185.8020604@quip.cz> <20080127112911.GA4024@voi.aagh.net> In-Reply-To: <20080127112911.GA4024@voi.aagh.net> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 12:25:57 -0000 Thomas Hurst wrote: > * Miroslav Lachman (000.fbsd@quip.cz) wrote: > > >>I found a painful performance problem with Apache + PHP 5 when >>open_basedir directive is enabled. > > > Looks like it's lstat()/readlink() overhead. I wrote a simple bit of > PHP similar to yours, but doing 10 * 1000 require "foo/%d/%d.php" calls > from the command line: > > -% time ktrace php main.php > Real: 0:09.61 CPU: 99.8% (3.725/5.885) Page: 0 Swap: 0 I/O: (0/0) Mem: > 13704 > > -% time ktrace php -d open_basedir='/home/freaky/openbasedir/foo' > main.php > Real: 0:16.21 CPU: 86.4% (8.185/5.840) Page: 0 Swap: 0 I/O: (0/0) Mem: > 13696 > > Without open_basedir, a simple script to parse the kdump syscall times > produces: > > lstat : 1.147s/70065 calls = 0.000s per call. Max=0.000s Min=0.0000s > fcntl : 0.408s/60007 calls = 0.000s per call. Max=0.000s Min=0.0000s > sigprocmask : 0.229s/40311 calls = 0.000s per call. Max=0.000s Min=0.0000s > open : 0.223s/10085 calls = 0.000s per call. Max=0.000s Min=0.0000s > > With open_basedir: > > lstat : 4.182s/270065 calls = 0.000s per call. Max=0.005s Min=0.0000s > readlink : 2.142s/10006 calls = 0.000s per call. Max=2.020s Min=0.0000s > fcntl : 0.421s/60007 calls = 0.000s per call. Max=0.002s Min=0.0000s > close : 0.295s/10085 calls = 0.000s per call. Max=0.115s Min=0.0000s > sigprocmask : 0.237s/40311 calls = 0.000s per call. Max=0.000s Min=0.0000s > open : 0.222s/10085 calls = 0.000s per call. Max=0.000s Min=0.0000s > > The top two syscalls seem to account for most of the 6.mumble seconds of > additional runtime; presumably these are much cheaper on Linux. I did some more research on saturday - test with old PHP 5 version 5.1.4 on FreeBSD 7.0 machine. It seems there is some significant change in PHP code. (on = with open_basedir / off = without open_basedir): # real webapplication FreeBSD 6.0 + PHP 5.1.4 on / off = 6.8 / 13.5 req/s FreeBSD 7.0 + PHP 5.2.5 on / off = 3.6 / 12.9 req/s FreeBSD 7.0 + PHP 5.1.4 on / off = 8.0 / 11.8 req/s # synthetic test FreeBSD 6.0 + PHP 5.1.4 on / off = 98.3 / 144.2 req/s FreeBSD 7.0 + PHP 5.2.5 on / off = 20.6 / 245.4 req/s FreeBSD 7.0 + PHP 5.1.4 on / off = 204.3 / 239.5 req/s It demonstrates my performance problem from year ago (also reported on this list 2007-02-03 with subject "PHP Performance problem after upgrade to 5.1.6 or 5.2.0"). It is still unsolved problem, but now I know it is influenced by open_basedir. As I reported a year ago - problem occured in 5.1.6 (maybe 5.1.5 - I never tried it, I jumped from 5.1.4 to 5.1.6 or newer). Can somebody look at PHP sources of those two versions and find the "bad change"? This was reported to PHP developers by me http://bugs.php.net/bug.php?id=38940 but marked as "bogus" with comment "We only support our own sources, not ports or patched source or binary packages." Is there any chance to fix this on FreeBSD side (patch in ports) or in PHP itself? I don't want to lose performance with each "security" upgrade of PHP. :( Miroslav Lachman From owner-freebsd-performance@FreeBSD.ORG Sun Jan 27 12:44:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63DAD16A41B for ; Sun, 27 Jan 2008 12:44:55 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from spork.qfe3.net (spork.qfe3.net [212.13.207.101]) by mx1.freebsd.org (Postfix) with ESMTP id 26DE513C468 for ; Sun, 27 Jan 2008 12:44:54 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from [81.104.123.28] (helo=voi.aagh.net) by spork.qfe3.net with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1JJ6sW-0003Zf-Pc; Sun, 27 Jan 2008 12:44:53 +0000 Received: from freaky by voi.aagh.net with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JJ6sW-000747-G3; Sun, 27 Jan 2008 12:44:52 +0000 Date: Sun, 27 Jan 2008 12:44:52 +0000 From: Thomas Hurst To: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <20080127124452.GA21973@voi.aagh.net> Mail-Followup-To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> <20080127112911.GA4024@voi.aagh.net> <479C7859.2060703@quip.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <479C7859.2060703@quip.cz> Organization: Not much. User-Agent: Mutt/1.5.17 (2007-11-01) Sender: Thomas Hurst Cc: freebsd-performance@freebsd.org Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 12:44:55 -0000 * Miroslav Lachman (000.fbsd@quip.cz) wrote: > As I reported a year ago - problem occured in 5.1.6 (maybe 5.1.5 - I > never tried it, I jumped from 5.1.4 to 5.1.6 or newer). Can somebody > look at PHP sources of those two versions and find the "bad change"? Handily I have 5.1.4 sources right next to my copy of trunk PHP5; main/fopen_wrappers.c php_check_specific_open_basedir() is about half the size there, and doesn't perform any readlink() calls, so it's probably vulnerable to escaping the basedir using symlinks, but considerably faster. -- Thomas 'Freaky' Hurst http://hur.st/ From owner-freebsd-performance@FreeBSD.ORG Sun Jan 27 14:54:02 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0543016A473 for ; Sun, 27 Jan 2008 14:54:02 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from mail.barafranca.com (mail.barafranca.com [67.213.67.47]) by mx1.freebsd.org (Postfix) with ESMTP id D043313C43E for ; Sun, 27 Jan 2008 14:54:01 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from localhost (unknown [172.16.100.24]) by mail.barafranca.com (Postfix) with ESMTP id E006C257B; Sun, 27 Jan 2008 14:38:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at barafranca.com Received: from mail.barafranca.com ([172.16.100.24]) by localhost (mail.barafranca.com [172.16.100.24]) (amavisd-new, port 10024) with ESMTP id inZSW2uRVBLh; Sun, 27 Jan 2008 14:38:26 +0000 (UTC) Received: from nexus.bsdlan.org (a213-22-38-76.cpe.netcabo.pt [213.22.38.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.barafranca.com (Postfix) with ESMTP id 242CF2575; Sun, 27 Jan 2008 14:38:24 +0000 (UTC) Message-ID: <479C97A3.2000009@barafranca.com> Date: Sun, 27 Jan 2008 14:39:31 +0000 From: Hugo Silva User-Agent: Thunderbird 2.0.0.6 (X11/20070816) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> In-Reply-To: <479B1185.8020604@quip.cz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Miroslav Lachman <000.fbsd@quip.cz> Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 14:54:02 -0000 Miroslav Lachman wrote: > Hi all, > > I found a painful performance problem with Apache + PHP 5 when > open_basedir directive is enabled. > Application performance drops by more than 50% with open_basedir enabled. > There is also significant disproportion between usr / sys CPU load. (sys > is much higher than usr) > I do not know if this is problem on FreeBSD side or in PHP code itself. > This problem exists on FreeBSD 6 and 7 (maybe older versions too - I did > not check). > I tested both shedulers on FreeBSD 7, no big difference (application > performance is little better with 4BSD). > Is there anyone with enough knowledge and time to look at it? > > This problem affects Mac OS X as well, but does not seem to appear on > Linux. > http://lists.apple.com/archives/macos-x-server/2006/Nov/msg00623.html I too have seen this, sys% is so MUCH higher and jumps from about 30% to 70% and higher all the time. In my case it's like 3% user time, 30-70% system time. I will try disabling open_basedir on one of the webservers and report back. Best regards, Hugo From owner-freebsd-performance@FreeBSD.ORG Sun Jan 27 16:48:12 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A75216A417 for ; Sun, 27 Jan 2008 16:48:12 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from mail.barafranca.com (mail.barafranca.com [67.213.67.47]) by mx1.freebsd.org (Postfix) with ESMTP id 60D6313C468 for ; Sun, 27 Jan 2008 16:48:12 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from localhost (unknown [172.16.100.24]) by mail.barafranca.com (Postfix) with ESMTP id D01A21BD5 for ; Sun, 27 Jan 2008 16:48:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at barafranca.com Received: from mail.barafranca.com ([172.16.100.24]) by localhost (mail.barafranca.com [172.16.100.24]) (amavisd-new, port 10024) with ESMTP id BlJnkIzpBiX6 for ; Sun, 27 Jan 2008 16:47:40 +0000 (UTC) Received: from [192.168.200.26] (a213-22-38-76.cpe.netcabo.pt [213.22.38.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.barafranca.com (Postfix) with ESMTP id 841B51BBB for ; Sun, 27 Jan 2008 16:47:39 +0000 (UTC) Message-ID: <479CB5AC.9060006@barafranca.com> Date: Sun, 27 Jan 2008 16:47:40 +0000 From: Hugo Silva User-Agent: Thunderbird 2.0.0.9 (X11/20071213) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> <479C97A3.2000009@barafranca.com> In-Reply-To: <479C97A3.2000009@barafranca.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jan 2008 16:48:12 -0000 Hugo Silva wrote: > Miroslav Lachman wrote: >> Hi all, >> >> I found a painful performance problem with Apache + PHP 5 when >> open_basedir directive is enabled. >> Application performance drops by more than 50% with open_basedir >> enabled. >> There is also significant disproportion between usr / sys CPU load. (sys >> is much higher than usr) >> I do not know if this is problem on FreeBSD side or in PHP code itself. >> This problem exists on FreeBSD 6 and 7 (maybe older versions too - I did >> not check). >> I tested both shedulers on FreeBSD 7, no big difference (application >> performance is little better with 4BSD). >> Is there anyone with enough knowledge and time to look at it? >> >> This problem affects Mac OS X as well, but does not seem to appear on >> Linux. >> http://lists.apple.com/archives/macos-x-server/2006/Nov/msg00623.html > > I too have seen this, sys% is so MUCH higher and jumps from about 30% > to 70% and higher all the time. In my case it's like 3% user time, > 30-70% system time. I will try disabling open_basedir on one of the > webservers and report back. > > Best regards, > > Hugo > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From 30-70% to 3%. What a difference! Hugo From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 01:10:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F20C116A41A for ; Mon, 28 Jan 2008 01:10:16 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from mail.barafranca.com (mail.barafranca.com [67.213.67.47]) by mx1.freebsd.org (Postfix) with ESMTP id C9DC713C44B for ; Mon, 28 Jan 2008 01:10:16 +0000 (UTC) (envelope-from hugo@barafranca.com) Received: from localhost (unknown [172.16.100.24]) by mail.barafranca.com (Postfix) with ESMTP id 0E43F1804 for ; Mon, 28 Jan 2008 01:10:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at barafranca.com Received: from mail.barafranca.com ([172.16.100.24]) by localhost (mail.barafranca.com [172.16.100.24]) (amavisd-new, port 10024) with ESMTP id XOwtnDUauqBi for ; Mon, 28 Jan 2008 01:09:48 +0000 (UTC) Received: from nexus.bsdlan.org (a213-22-38-76.cpe.netcabo.pt [213.22.38.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.barafranca.com (Postfix) with ESMTP id B58301801 for ; Mon, 28 Jan 2008 01:09:47 +0000 (UTC) Message-ID: <479D2B9E.20908@barafranca.com> Date: Mon, 28 Jan 2008 01:10:54 +0000 From: Hugo Silva User-Agent: Thunderbird 2.0.0.6 (X11/20070816) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> <479C97A3.2000009@barafranca.com> <479CB5AC.9060006@barafranca.com> In-Reply-To: <479CB5AC.9060006@barafranca.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 01:10:17 -0000 Hugo Silva wrote: > Hugo Silva wrote: >> Miroslav Lachman wrote: >>> Hi all, >>> >>> I found a painful performance problem with Apache + PHP 5 when >>> open_basedir directive is enabled. >>> Application performance drops by more than 50% with open_basedir >>> enabled. >>> There is also significant disproportion between usr / sys CPU load. >>> (sys >>> is much higher than usr) >>> I do not know if this is problem on FreeBSD side or in PHP code itself. >>> This problem exists on FreeBSD 6 and 7 (maybe older versions too - I >>> did >>> not check). >>> I tested both shedulers on FreeBSD 7, no big difference (application >>> performance is little better with 4BSD). >>> Is there anyone with enough knowledge and time to look at it? >>> >>> This problem affects Mac OS X as well, but does not seem to appear >>> on Linux. >>> http://lists.apple.com/archives/macos-x-server/2006/Nov/msg00623.html >> >> I too have seen this, sys% is so MUCH higher and jumps from about 30% >> to 70% and higher all the time. In my case it's like 3% user time, >> 30-70% system time. I will try disabling open_basedir on one of the >> webservers and report back. >> >> Best regards, >> >> Hugo >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to >> "freebsd-performance-unsubscribe@freebsd.org" > > From 30-70% to 3%. What a difference! > > Hugo > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" http://bugs.php.net/bug.php?id=43946 From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 08:53:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 689C616A41B for ; Mon, 28 Jan 2008 08:53:05 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id E9AC013C4EC for ; Mon, 28 Jan 2008 08:53:04 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.14) with ESMTPA id 215120382; Mon, 28 Jan 2008 10:52:03 +0300 Message-ID: <479D89C9.7060300@chistydom.ru> Date: Mon, 28 Jan 2008 10:52:41 +0300 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Miroslav Lachman <000.fbsd@quip.cz> References: <479B1185.8020604@quip.cz> In-Reply-To: <479B1185.8020604@quip.cz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 28 Jan 2008 12:25:54 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 08:53:05 -0000 The problem is that concurrent lstat()'s are very slow on FreeBSD now. Possibly you can decrease the number of lstat() calls by tuning realpath cache size in PHP. Just add "realpath_cache_size=512k" to php.ini. However I'm not sure this cache is used in open_basedir. See the following thread for details: http://lists.freebsd.org/pipermail/freebsd-stable/2007-November/038371.html Miroslav Lachman wrote: > Hi all, > > I found a painful performance problem with Apache + PHP 5 when > open_basedir directive is enabled. > Application performance drops by more than 50% with open_basedir enabled. > There is also significant disproportion between usr / sys CPU load. (sys > is much higher than usr) > I do not know if this is problem on FreeBSD side or in PHP code itself. > This problem exists on FreeBSD 6 and 7 (maybe older versions too - I did > not check). > I tested both shedulers on FreeBSD 7, no big difference (application > performance is little better with 4BSD). > Is there anyone with enough knowledge and time to look at it? From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 13:37:46 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A14216A41A for ; Mon, 28 Jan 2008 13:37:46 +0000 (UTC) (envelope-from fbsd-performance@mawer.org) Received: from outbound.icp-qv1-irony-out1.iinet.net.au (outbound.icp-qv1-irony-out1.iinet.net.au [203.59.1.108]) by mx1.freebsd.org (Postfix) with ESMTP id F263313C4E8 for ; Mon, 28 Jan 2008 13:37:45 +0000 (UTC) (envelope-from fbsd-performance@mawer.org) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAGpnnUd8qoGx/2dsb2JhbAAIqms X-IronPort-AV: E=Sophos;i="4.25,260,1199631600"; d="scan'208";a="279790971" Received: from unknown (HELO [10.4.1.1]) ([124.170.129.177]) by outbound.icp-qv1-irony-out1.iinet.net.au with ESMTP; 28 Jan 2008 22:27:41 +0900 Message-ID: <479DD94C.7010409@mawer.org> Date: Tue, 29 Jan 2008 00:31:56 +1100 From: Antony Mawer User-Agent: Thunderbird 1.5.0.13 (Windows/20070809) MIME-Version: 1.0 To: Alexey Popov References: <479B1185.8020604@quip.cz> <479D89C9.7060300@chistydom.ru> In-Reply-To: <479D89C9.7060300@chistydom.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Miroslav Lachman <000.fbsd@quip.cz> Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 13:37:46 -0000 On 28/01/2008 6:52 PM, Alexey Popov wrote: > The problem is that concurrent lstat()'s are very slow on FreeBSD now. > > Possibly you can decrease the number of lstat() calls by tuning realpath > cache size in PHP. Just add "realpath_cache_size=512k" to php.ini. > However I'm not sure this cache is used in open_basedir. > > See the following thread for details: > http://lists.freebsd.org/pipermail/freebsd-stable/2007-November/038371.html ... so how does one go about profiling lstat() to find out where the source of the slowness originates from? Is pmc profiling the way to do this? The last thread on this topic (referenced above) seemed to point the finger at the default value of vfs.ufs.dirhash_maxmem being too small... which would suggest this is an inherit limitation in UFS(2?) and/or its default tuning. If you try increasing the amount of memory allocated for dirhash, does this improve performance at all? eg: # sysctl vfs.ufs.dirhash_maxmem=10240000 vfs.ufs.dirhash_maxmem: 2097152 -> 10240000 ... and then re-run the benchmarks to see if this makes any noticeable difference. The fact that it was indicated that Mac OS X also suffers the same problems suggests this may be a VFS issue rather than UFS specific...? --Antony From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 14:23:47 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B07D16A46C for ; Mon, 28 Jan 2008 14:23:47 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from mail.sweeplist.com (mail.sweeplist.com [77.93.199.51]) by mx1.freebsd.org (Postfix) with ESMTP id 0D87613C455 for ; Mon, 28 Jan 2008 14:23:46 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from [192.168.1.2] (r3a200.net.upc.cz [213.220.192.200]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sweeplist.com (Postfix) with ESMTPSA id 1349367808; Mon, 28 Jan 2008 15:23:46 +0100 (CET) Message-ID: <479DE578.7060202@quip.cz> Date: Mon, 28 Jan 2008 15:23:52 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <479B1185.8020604@quip.cz> <479D89C9.7060300@chistydom.ru> <479DD94C.7010409@mawer.org> In-Reply-To: <479DD94C.7010409@mawer.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: Alexey Popov , Antony Mawer Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 14:23:47 -0000 Antony Mawer wrote: > On 28/01/2008 6:52 PM, Alexey Popov wrote: > >> The problem is that concurrent lstat()'s are very slow on FreeBSD now. >> >> Possibly you can decrease the number of lstat() calls by tuning >> realpath cache size in PHP. Just add "realpath_cache_size=512k" to >> php.ini. However I'm not sure this cache is used in open_basedir. >> >> See the following thread for details: >> http://lists.freebsd.org/pipermail/freebsd-stable/2007-November/038371.html >> > > > ... so how does one go about profiling lstat() to find out where the > source of the slowness originates from? Is pmc profiling the way to do > this? > > The last thread on this topic (referenced above) seemed to point the > finger at the default value of vfs.ufs.dirhash_maxmem being too small... > which would suggest this is an inherit limitation in UFS(2?) and/or its > default tuning. > > If you try increasing the amount of memory allocated for dirhash, does > this improve performance at all? eg: > > # sysctl vfs.ufs.dirhash_maxmem=10240000 > vfs.ufs.dirhash_maxmem: 2097152 -> 10240000 > > ... and then re-run the benchmarks to see if this makes any noticeable > difference. The fact that it was indicated that Mac OS X also suffers > the same problems suggests this may be a VFS issue rather than UFS > specific...? I tried sysctl vfs.ufs.dirhash_maxmem=10240000 and realpath_cache_size=512k in php.ini and sysctl vfs.lookup_shared=1 but all without any significant impact on performance with open_basedir enabled. I also tried this patch for vfs_cache http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_cache.c.diff?r1=1.114;r2=1.115 but again without success. top shows following CPU usage for synthetic test with PHP 5.2.5 on 7.0-BETA4 with SCHED_ULE an vfs_cache patch + both sysctl tunables and php.ini directive: CPU states: 8.1% user, 0.0% nice, 88.6% system, 0.2% interrupt, 3.2% idle Does somebody have any other ideas? Miroslav Lachman From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 15:28:42 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DF9616A417 for ; Mon, 28 Jan 2008 15:28:42 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.241]) by mx1.freebsd.org (Postfix) with ESMTP id D034813C457 for ; Mon, 28 Jan 2008 15:28:41 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so448045anc.13 for ; Mon, 28 Jan 2008 07:28:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; bh=ZoKXLTDuoGsZTT8FNhhCRTUNnzHdBeOHNC6wjqtR7mU=; b=KDTyfbBUe4Z+08blNl9aPUn5m4SoQ6NiM6oG/i7zG+1IFMXeoyNWCikwZuqAwsTwVVSvs9Kp/h7GTYWuTmzlSh7HkoAwy9BbEnQJpP3n36w5ZqjxPxSpnwrx5+qKwgM3dqXZy+bpPEF/QmVPOLl8aVc5bS6/nSb0lHAsd8ocC3g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition; b=Nm3abUHt3oz8TGh4SyqOoS7IlCGYOychg+TK5ec/uVaL/L1YQOKNRPCnCJ6D/ekPnp+sSNywlroaDjVM3bOB5NRZLMXpVgwuRTJelE5nv7mleREpg/aKLJb1psZFkvV3QgPY7HCe6lMqLZWHUg8xv72ugEx7SbCP4qBfiuWon5Y= Received: by 10.100.216.3 with SMTP id o3mr12065178ang.95.1201532471204; Mon, 28 Jan 2008 07:01:11 -0800 (PST) Received: by 10.100.242.10 with HTTP; Mon, 28 Jan 2008 07:01:11 -0800 (PST) Message-ID: Date: Mon, 28 Jan 2008 16:01:11 +0100 From: "Claus Guttesen" To: "FreeBSD Mailing Lists" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 15:28:42 -0000 Hi. I have a HP DL360 G5 with a p800 controller with 512 bbwc and a msa70 cabinet with eight 15K rpm sas-disks in raid 1+0. I installed FreeBSD 7.0 stable and ubuntu 7.10 server using postgresql 8.2.5 (from ports on FreeBSD and as an install-option on ubuntu). Both releases are amd64. Postgresql on FreeBSD was compiled with WITH_THREADSAFE=true (/var/db/ports/postgresql82/options). The server is a two x quad-core @ 2.33 GHz with 16 GB ram. I installed RC1 and csup'ed to latest FreeBSD 7 stable and recompiled world and kernel. Removed options and devices I don't need like msdos, ntfs, nfs. Ubuntu is a stock install, ie. kernel-upgrade. Kernel does not have WITNESS etc. Then I installed and ran sysbench on postgresql using these settings: Postgresql.conf is the stock-configuration as created when running initdb except I raised shared_buffers = 128MB on both FreeBSD and Ubuntu (and raised sysctl-values as well to accomodate the larger shared_buffers). sysbench --test=oltp --pgsql-user=claus --oltp-table-size=100000 prepare After that I ran sysbench using this loop: run=5 clients=50 for ((a=10; a<=$clients; a=a+2)) do for ((b=1; b<=$run; b++)) do psql -d sbtest -h localhost -U claus -c "vacuum analyse;" sysbench --test=oltp --num-threads=${a} --pgsql-user=claus --pgsql-host="" --max-time=300 --max-requests=10000 --oltp-table-size=100000 run >> sysbench-clients-${a} done done Ubuntu 7.10: grep "transactions:" sysbench-clients-24|sort transactions: 10000 (2354.49 per sec.) transactions: 10001 (2126.28 per sec.) transactions: 10001 (2215.52 per sec.) transactions: 10001 (2236.03 per sec.) FreeBSD 7.0 stable as of Jan. 28'th: grep "transactions:" sysbench-clients-24|sort transactions: 10001 (1600.36 per sec.) transactions: 10002 (1963.95 per sec.) transactions: 10005 (1973.17 per sec.) In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs I could try on FreeBSD? -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 15:15:02 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A220F16A418 for ; Mon, 28 Jan 2008 15:15:02 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id 2FFE313C4CC for ; Mon, 28 Jan 2008 15:15:01 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.14) with ESMTPA id 215272070; Mon, 28 Jan 2008 18:14:03 +0300 Message-ID: <479DF162.3080803@chistydom.ru> Date: Mon, 28 Jan 2008 18:14:42 +0300 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Miroslav Lachman <000.fbsd@quip.cz> References: <479B1185.8020604@quip.cz> <479D89C9.7060300@chistydom.ru> <479DD94C.7010409@mawer.org> <479DE578.7060202@quip.cz> In-Reply-To: <479DE578.7060202@quip.cz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 28 Jan 2008 15:32:41 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: PHP with open_basedir performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 15:15:02 -0000 Hi Miroslav Lachman wrote: > I tried sysctl vfs.ufs.dirhash_maxmem=10240000 and > realpath_cache_size=512k in php.ini and sysctl vfs.lookup_shared=1 > but all without any significant impact on performance with open_basedir > enabled. > CPU states: 8.1% user, 0.0% nice, 88.6% system, 0.2% interrupt, 3.2% > idle > Does somebody have any other ideas? Here's the patch helped me in the similar situation: http://lists.freebsd.org/pipermail/freebsd-stable/2007-November/038449.html If it also does not help, I think LOCK_PROFILING(9) and pmc stats ( http://freebsd.rambler.ru/bsdmail/freebsd-current_2006/msg01581.html ) would be useful to see what exactly is slow. With best regards, Alexey Popov From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 17:25:33 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE06216A417 for ; Mon, 28 Jan 2008 17:25:33 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 35FB213C442 for ; Mon, 28 Jan 2008 17:25:32 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JJXja-0006Ff-1I for freebsd-performance@freebsd.org; Mon, 28 Jan 2008 17:25:26 +0000 Received: from 89-172-47-154.adsl.net.t-com.hr ([89.172.47.154]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 28 Jan 2008 17:25:26 +0000 Received: from ivoras by 89-172-47-154.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 28 Jan 2008 17:25:26 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 28 Jan 2008 18:25:13 +0100 Lines: 51 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigDE192207C6F41BC1F0AD8AE2" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 89-172-47-154.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 17:25:33 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigDE192207C6F41BC1F0AD8AE2 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Claus Guttesen wrote: > Ubuntu 7.10: >=20 > grep "transactions:" sysbench-clients-24|sort > transactions: 10000 (2354.49 per sec.) > transactions: 10001 (2126.28 per sec.) > transactions: 10001 (2215.52 per sec.) > transactions: 10001 (2236.03 per sec.) >=20 > FreeBSD 7.0 stable as of Jan. 28'th: >=20 > grep "transactions:" sysbench-clients-24|sort > transactions: 10001 (1600.36 per sec.) > transactions: 10002 (1963.95 per sec.) > transactions: 10005 (1973.17 per sec.) >=20 > In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs > I could try on FreeBSD? I think the excellent results Kris got with FreeBSD were significantly=20 helped by patching postgresql to remove setproctitle(). Other than that, = from the sysbench line I see this is OLTP benchmark which should mean=20 a lot of write transactions, and I've consistently seen much better file = system write performance on Linux than on FreeBSD. No tuning can help her= e. --------------enigDE192207C6F41BC1F0AD8AE2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHng/5ldnAQVacBcgRAmSxAKDJMMxOowRxmfBd1wM4vavU2As2XACglmdd xBUMPGxWnmOWIxVcjNKkYaY= =RY6+ -----END PGP SIGNATURE----- --------------enigDE192207C6F41BC1F0AD8AE2-- From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 18:47:20 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 197A316A421 for ; Mon, 28 Jan 2008 18:47:20 +0000 (UTC) (envelope-from darcyb@commandprompt.com) Received: from lists.commandprompt.com (host-159.commandprompt.net [207.173.203.159]) by mx1.freebsd.org (Postfix) with ESMTP id ED52513C4FB for ; Mon, 28 Jan 2008 18:47:19 +0000 (UTC) (envelope-from darcyb@commandprompt.com) Received: from dbitech.homenet.dbitech.bc.ca (S010600a08330057a.ok.shawcable.net [24.67.72.171]) (authenticated bits=0) by lists.commandprompt.com (8.13.8/8.13.8) with ESMTP id m0SIObOH009862 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 28 Jan 2008 10:24:39 -0800 From: Darcy Buskermolen Organization: Command Prompt To: freebsd-performance@freebsd.org Date: Mon, 28 Jan 2008 10:24:10 -0800 User-Agent: KMail/1.9.7 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200801281024.11571.darcyb@commandprompt.com> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (lists.commandprompt.com [207.173.203.159]); Mon, 28 Jan 2008 10:24:39 -0800 (PST) X-Mailman-Approved-At: Mon, 28 Jan 2008 19:32:26 +0000 Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 18:47:20 -0000 On Monday 28 January 2008 09:25:13 Ivan Voras wrote: > Claus Guttesen wrote: > > Ubuntu 7.10: > > > > grep "transactions:" sysbench-clients-24|sort > > transactions: 10000 (2354.49 per sec.) > > transactions: 10001 (2126.28 per sec.) > > transactions: 10001 (2215.52 per sec.) > > transactions: 10001 (2236.03 per sec.) > > > > FreeBSD 7.0 stable as of Jan. 28'th: > > > > grep "transactions:" sysbench-clients-24|sort > > transactions: 10001 (1600.36 per sec.) > > transactions: 10002 (1963.95 per sec.) > > transactions: 10005 (1973.17 per sec.) > > > > In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs > > I could try on FreeBSD? > > I think the excellent results Kris got with FreeBSD were significantly > helped by patching postgresql to remove setproctitle(). You don;t need to patch postgresql for that, all you need to do is turn that off. update_process_title = off in postgresql.conf and then restart the daemon. > Other than that, > from the sysbench line I see this is OLTP benchmark which should mean > a lot of write transactions, and I've consistently seen much better file > system write performance on Linux than on FreeBSD. No tuning can help here. -- Darcy Buskermolen Command Prompt, Inc. +1.503.667.4564 X 102 http://www.commandprompt.com/ PostgreSQL solutions since 1997 From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 20:21:03 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D77EC16A420 for ; Mon, 28 Jan 2008 20:21:03 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.250]) by mx1.freebsd.org (Postfix) with ESMTP id 9080B13C461 for ; Mon, 28 Jan 2008 20:21:03 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so479618anc.13 for ; Mon, 28 Jan 2008 12:21:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=LZXgiIrAm5Z28RHM8oVkGrbq/be7Nz6M6tOaReufpeQ=; b=hjrxA67HB2JPpySlgOs4ev3IXwmZU954wbXuUK5T5mdQe7QsZFlwxBvJ529lEpJ6nNk//tXxhLKdaLMcll82brKCwnTGUa5E3psz3jvZEmPBzAqmR95VUI6oFJa8x41J69MoyYcH+fAfpb1BM9N6n4RwDSSwSbyUrTvPC/KJn9U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=pv/8g8FjWShGzEh8dxm0SvgkNGGEGuXCsi+lCOY3AEf8H4JUsZe9ksLjvhU58vygjAZLBdMQYKqOmTKxtozMjLvX2PCUM8f+VI5GRqvr2i8w3aCc9SFvidKmfFkedOUQldzaZwb5pZal90wHCRQnRf53nUTkw77P+HPD/VqiCDQ= Received: by 10.100.123.4 with SMTP id v4mr12900878anc.14.1201551663033; Mon, 28 Jan 2008 12:21:03 -0800 (PST) Received: by 10.100.242.10 with HTTP; Mon, 28 Jan 2008 12:21:03 -0800 (PST) Message-ID: Date: Mon, 28 Jan 2008 21:21:03 +0100 From: "Claus Guttesen" To: "Darcy Buskermolen" In-Reply-To: <200801281024.11571.darcyb@commandprompt.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200801281024.11571.darcyb@commandprompt.com> Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 20:21:03 -0000 > > > Ubuntu 7.10: > > > > > > grep "transactions:" sysbench-clients-24|sort > > > transactions: 10000 (2354.49 per sec.) > > > transactions: 10001 (2126.28 per sec.) > > > transactions: 10001 (2215.52 per sec.) > > > transactions: 10001 (2236.03 per sec.) > > > > > > FreeBSD 7.0 stable as of Jan. 28'th: > > > > > > grep "transactions:" sysbench-clients-24|sort > > > transactions: 10001 (1600.36 per sec.) > > > transactions: 10002 (1963.95 per sec.) > > > transactions: 10005 (1973.17 per sec.) > > > > > > In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs > > > I could try on FreeBSD? > > > > I think the excellent results Kris got with FreeBSD were significantly > > helped by patching postgresql to remove setproctitle(). > > You don;t need to patch postgresql for that, all you need to do is turn that > off. > > update_process_title = off in postgresql.conf and then restart the daemon. I found the setting and set it to off but no real difference in performance. > > from the sysbench line I see this is OLTP benchmark which should mean > > a lot of write transactions, and I've consistently seen much better file > > system write performance on Linux than on FreeBSD. No tuning can help here. Yes, that is correct. I wanted to conduct a r/w test. But if it's down to the fs itself I will just leave it atm. I will probably deploy the server on FreeBSD anyway since we probably won't reach that many writes in the foreseable future and FreeBSD is what I do best. Will zfs be able to achieve better performance? I guess that ufs2 will remain more or less in the state it is in now. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 20:34:39 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 950E716A41B for ; Mon, 28 Jan 2008 20:34:39 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 927A313C45B; Mon, 28 Jan 2008 20:34:38 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <479E3C5E.1070405@FreeBSD.org> Date: Mon, 28 Jan 2008 21:34:38 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Claus Guttesen References: <200801281024.11571.darcyb@commandprompt.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Darcy Buskermolen , freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 20:34:39 -0000 Claus Guttesen wrote: >>>> Ubuntu 7.10: >>>> >>>> grep "transactions:" sysbench-clients-24|sort >>>> transactions: 10000 (2354.49 per sec.) >>>> transactions: 10001 (2126.28 per sec.) >>>> transactions: 10001 (2215.52 per sec.) >>>> transactions: 10001 (2236.03 per sec.) >>>> >>>> FreeBSD 7.0 stable as of Jan. 28'th: >>>> >>>> grep "transactions:" sysbench-clients-24|sort >>>> transactions: 10001 (1600.36 per sec.) >>>> transactions: 10002 (1963.95 per sec.) >>>> transactions: 10005 (1973.17 per sec.) >>>> >>>> In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs >>>> I could try on FreeBSD? >>> I think the excellent results Kris got with FreeBSD were significantly >>> helped by patching postgresql to remove setproctitle(). >> You don;t need to patch postgresql for that, all you need to do is turn that >> off. >> >> update_process_title = off in postgresql.conf and then restart the daemon. > > I found the setting and set it to off but no real difference in performance. > >>> from the sysbench line I see this is OLTP benchmark which should mean >>> a lot of write transactions, and I've consistently seen much better file >>> system write performance on Linux than on FreeBSD. No tuning can help here. > > Yes, that is correct. I wanted to conduct a r/w test. But if it's down > to the fs itself I will just leave it atm. I will probably deploy the > server on FreeBSD anyway since we probably won't reach that many > writes in the foreseable future and FreeBSD is what I do best. > > Will zfs be able to achieve better performance? I guess that ufs2 will > remain more or less in the state it is in now. > I went through this in detail in a thread on -stable recently (Subject: Performance!). Rather than me going over all of this again, can you please read that thread in detail and get back to me once you have applied all of the discussion there to your case. Kris From owner-freebsd-performance@FreeBSD.ORG Mon Jan 28 20:46:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96C5116A417 for ; Mon, 28 Jan 2008 20:46:55 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.248]) by mx1.freebsd.org (Postfix) with ESMTP id 48EB413C47E for ; Mon, 28 Jan 2008 20:46:55 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so482542anc.13 for ; Mon, 28 Jan 2008 12:46:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=QB60fgUhS8h5Gsz/VO0z9KtCPn8C2LEeAyPgZyu1WpU=; b=oCUhl2JK5GMffzTUY5zmHf8wqri9HUjLGDkgkEGsWAKbVf5B0u0s1Psnr738BLAqa+rjroZYt7NKDJw13BBi0rq5GAnLPCXxgtSz8A9fGrHYgr4L6ZKQ+c63cI7vnpXk+82HDTbPT3H7XK4euKARM3SJFniJvzeNXVknBot5clg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=G/XpDKBtYk2LQGcNzdNkzrHu6l7TtbrDNkQbCBc82+OUkAT4uDKqqrvt4vxgbZn0yVZqis79iSm/jGmdgqAqY4320eerbKLJXSGeu5LzG4Kkf5vUd8RTTVXI6LWCdLmnDVeCg/0NpdbzHpDz8bjF67+o8V5C5561HVDkHGbwKqk= Received: by 10.100.216.3 with SMTP id o3mr12860833ang.86.1201553214468; Mon, 28 Jan 2008 12:46:54 -0800 (PST) Received: by 10.100.242.10 with HTTP; Mon, 28 Jan 2008 12:46:54 -0800 (PST) Message-ID: Date: Mon, 28 Jan 2008 21:46:54 +0100 From: "Claus Guttesen" To: "Kris Kennaway" In-Reply-To: <479E3C5E.1070405@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> Cc: Darcy Buskermolen , freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jan 2008 20:46:55 -0000 > >>>> Ubuntu 7.10: > >>>> > >>>> grep "transactions:" sysbench-clients-24|sort > >>>> transactions: 10000 (2354.49 per sec.) > >>>> transactions: 10001 (2126.28 per sec.) > >>>> transactions: 10001 (2215.52 per sec.) > >>>> transactions: 10001 (2236.03 per sec.) > >>>> > >>>> FreeBSD 7.0 stable as of Jan. 28'th: > >>>> > >>>> grep "transactions:" sysbench-clients-24|sort > >>>> transactions: 10001 (1600.36 per sec.) > >>>> transactions: 10002 (1963.95 per sec.) > >>>> transactions: 10005 (1973.17 per sec.) > >>>> > >>>> In other runs FreeBSD also seems to trail Ubuntu. Are there any knobs > >>>> I could try on FreeBSD? > >>> I think the excellent results Kris got with FreeBSD were significantly > >>> helped by patching postgresql to remove setproctitle(). > >> You don;t need to patch postgresql for that, all you need to do is turn that > >> off. > >> > >> update_process_title = off in postgresql.conf and then restart the daemon. > > > > I found the setting and set it to off but no real difference in performance. > > > >>> from the sysbench line I see this is OLTP benchmark which should mean > >>> a lot of write transactions, and I've consistently seen much better file > >>> system write performance on Linux than on FreeBSD. No tuning can help here. > > > > Yes, that is correct. I wanted to conduct a r/w test. But if it's down > > to the fs itself I will just leave it atm. I will probably deploy the > > server on FreeBSD anyway since we probably won't reach that many > > writes in the foreseable future and FreeBSD is what I do best. > > > > Will zfs be able to achieve better performance? I guess that ufs2 will > > remain more or less in the state it is in now. I wouldn't deploy zfs on a prod. db-server so this is just to hear if some had tried zfs (with compression enabled). > > I went through this in detail in a thread on -stable recently (Subject: > Performance!). Rather than me going over all of this again, can you > please read that thread in detail and get back to me once you have > applied all of the discussion there to your case. I had (allready) saved the thread in my mail-account so I could look it up before I started testing. :-) So I compiled postgresql with the option WITH_THREADSAFE=true and used sysbench with --pgsql-host="" . As pointed out by Ivan my test also involved r/w whereas the thread you (probably) mention at http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/e224cd4f76e9ec2d is a read-only test. I forgot to mention in my first post that I'm using ULE. The p800 controller has a (factory set) 25/75 read/write cache ratio. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 09:32:52 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB7B716A41A for ; Tue, 29 Jan 2008 09:32:52 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 7133C13C43E for ; Tue, 29 Jan 2008 09:32:52 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JJmph-00073U-VZ for freebsd-performance@freebsd.org; Tue, 29 Jan 2008 09:32:45 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 29 Jan 2008 09:32:45 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 29 Jan 2008 09:32:45 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Tue, 29 Jan 2008 10:32:54 +0100 Lines: 39 Message-ID: References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig7F31BBEEDDC685A4EC4BA658" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 09:32:53 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig7F31BBEEDDC685A4EC4BA658 Claus Guttesen wrote: > I had (allready) saved the thread in my mail-account so I could look > it up before I started testing. :-) So I compiled postgresql with the > option WITH_THREADSAFE=3Dtrue and used sysbench with --pgsql-host=3D"" = =2E > As pointed out by Ivan my test also involved r/w whereas the thread > you (probably) mention at > http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thr= ead/e224cd4f76e9ec2d > is a read-only test. >=20 > I forgot to mention in my first post that I'm using ULE. The p800 > controller has a (factory set) 25/75 read/write cache ratio. There's maybe one additional thing: do you dual-boot Linux and FreeBSD? If so, you'll need to set up a separate additional partition for the database, instead of benchmarking it with the file systems used by the OS, because different areas of the drive(s) have different performance - you can verify this with diskinfo -t. --------------enig7F31BBEEDDC685A4EC4BA658 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHnvLTldnAQVacBcgRAq/eAKC74EcOl2Xs9whdrhauOSxNtx+35wCgobAb gh6afuw0X2ED5kPd7aRGm98= =Mvec -----END PGP SIGNATURE----- --------------enig7F31BBEEDDC685A4EC4BA658-- From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 09:47:13 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 594D416A420 for ; Tue, 29 Jan 2008 09:47:13 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.243]) by mx1.freebsd.org (Postfix) with ESMTP id 1148D13C4E5 for ; Tue, 29 Jan 2008 09:47:12 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so541961anc.13 for ; Tue, 29 Jan 2008 01:47:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=ng39Mc4/dxOO9oWTgEWSbJjEbmYUa5+5aN4fMI4wKs8=; b=LvonHiMZHPPCUX+UFrYfHtYs3jDzXWizl5sxwAgL0NnCYkDzC3S75zZiNOf+7SsYaAiwsnZjxAOhBUhPj5o3yENonA/MGcvD9UAeDs9cdNJruE0SzRjppDs4CCxwrwQAL/+q0nLcVbNY0Ui5b9fZ8+Wx6dMdLwx0afPxsyva6Qg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=A5B+edldDJnGA8+sYI8rgZ1k6ZFmSmwFz8WuzeTNmrTCeXLhNnrFsM1VgRMtLZzBsLUIqoVPV25Qt37zZobTmnQP7I2d/g1mJlHkJreCZA7w/X6QT2iU0uzZFp1YBy8ShZymUa9Z7TnmmhvVXR3Tkm0HQv8iqm0EEzN3mBsXpew= Received: by 10.100.201.16 with SMTP id y16mr14312004anf.30.1201600007813; Tue, 29 Jan 2008 01:46:47 -0800 (PST) Received: by 10.100.242.10 with HTTP; Tue, 29 Jan 2008 01:46:47 -0800 (PST) Message-ID: Date: Tue, 29 Jan 2008 10:46:47 +0100 From: "Claus Guttesen" To: "Ivan Voras" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 09:47:13 -0000 > > I forgot to mention in my first post that I'm using ULE. The p800 > > controller has a (factory set) 25/75 read/write cache ratio. > > There's maybe one additional thing: do you dual-boot Linux and FreeBSD? > If so, you'll need to set up a separate additional partition for the > database, instead of benchmarking it with the file systems used by the > OS, because different areas of the drive(s) have different performance - > you can verify this with diskinfo -t. I installed FreeBSD onto a boot-partition (p400i-controller) and used the external storage (p800) as database-partition (eight 15K rpm sas-disks in raid 1+0). Same with Ubuntu. When I re-installed FreeBSD and ubuntu I wiped and formatted the previous partitions. Ubuntu used ext3 which I guess is default fs. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 10:59:58 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 270EB16A418 for ; Tue, 29 Jan 2008 10:59:58 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id DC3BF13C465 for ; Tue, 29 Jan 2008 10:59:57 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from [127.0.0.1] (fw.axelero.hu [195.228.243.120]) by people.fsn.hu (Postfix) with ESMTP id 7D6FE6D832; Tue, 29 Jan 2008 11:40:55 +0100 (CET) Message-ID: <479F02A7.9020607@fsn.hu> Date: Tue, 29 Jan 2008 11:40:39 +0100 From: Attila Nagy User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 10:59:58 -0000 On 2008.01.28. 19:21, JINMEI Tatuya / 神жЋйЃ”哉 wrote: > At Mon, 28 Jan 2008 17:10:28 +0100, > Attila Nagy wrote: > > >>> If you have time, could you rebuild named as follows >>> >>> % STD_CDEFINES='-DLRU_DEBUG' ./configure; make >>> >>> and try again? This won't solve the problem, but provide more >>> detailed log messages tracking the memory usage of the cache. Make >>> sure threads are disabled (this debug feature is currently >>> thread-unsafe and trigger an assertion failure when used with multiple >>> threads). >>> >>> >> Without threading I don't see this effect, the memory usage stops at a >> sane limit and it's size can be affected by setting the max-cache-size >> option. >> >> I don't think you would gain anything usable with that, am I right? >> > > Right. Can you try a simpler patch that focuses on the memory usage > status and works with threads? If so, I'll write one and send it to > you. > Of course. The machines are diskless, so writing larger log files directly is not an easy task. (syslog is ok) Thanks. ps: I have an other problem. I've recently switched from a last year 6-STABLE to 7-STABLE and got pretty bad results on the same machine with the same bind (9.4). The graphs are here: http://picasaweb.google.com/nagy.attila/20080129Fbsd6vs7Bind The interesting part (from when the comments are valid) starts at around the half of the picture. You can see that on FreeBSD 6, the CPU load is pretty much good, but on 7, both the userspace and the kernelspace activity grows significantly. I've used libthr on 6, and it is the default on 7 too. bind is threaded. I use ISC_INTERNAL_MALLOC, but the effect is the same without it. I've cc-ed the freebsd-performance, I guess there are several other ppl, who is interested in bind performance on this OS. From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 14:27:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D336016A420 for ; Tue, 29 Jan 2008 14:27:36 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.245]) by mx1.freebsd.org (Postfix) with ESMTP id 8B02513C469 for ; Tue, 29 Jan 2008 14:27:36 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so565796anc.13 for ; Tue, 29 Jan 2008 06:27:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=BtSgU9mPsmSjZ4b6nTbMtjLC1mE8Kwn2Uje6x2aM83c=; b=XTQqAOZq4lOCZEEUJR1pkuB+6RdTfWfgeL6LXWxkXoqYj+9zTgXTP+04H+aV6WRdThZNMDSRU/PLcehQELSAnUnRm7psDfJ1PC5ymWsOc9zyTcVLLvUU6766qMANppT0LD1+57Vshm+Zw1IEaa8cJN1QOC0kXa4+gwgPYY9DJVc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=e2fulleActKXFiaQ4s1/nfwDlOlTe6WYvJxrsRQ5L0GO90M8RhJpPXtQWShQT877jQFRoMOnZTU1eoulMzU/kJMKa0lVQjbdnaGCYNpqB5GNCSElQrABc1MWL3fDRmBe7WxTs0doODY5GMfM/6PeIkfT3BJ35rfqv4ewZcp7J6g= Received: by 10.100.57.6 with SMTP id f6mr14752207ana.113.1201616831119; Tue, 29 Jan 2008 06:27:11 -0800 (PST) Received: by 10.100.242.10 with HTTP; Tue, 29 Jan 2008 06:27:11 -0800 (PST) Message-ID: Date: Tue, 29 Jan 2008 15:27:11 +0100 From: "Claus Guttesen" To: "Mike Tancsa" In-Reply-To: <200801291418.m0TEIbsw076723@lava.sentex.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <200801291418.m0TEIbsw076723@lava.sentex.ca> Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 14:27:36 -0000 > >I had (allready) saved the thread in my mail-account so I could look > >it up before I started testing. :-) So I compiled postgresql with the > >option WITH_THREADSAFE=true and used sysbench with --pgsql-host="" . > >As pointed out by Ivan my test also involved r/w whereas the thread > >you (probably) mention at > >http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/e224cd4f76e9ec2d > >is a read-only test. > > > >I forgot to mention in my first post that I'm using ULE. The p800 > >controller has a (factory set) 25/75 read/write cache ratio. > > > I am still going through my own testing. One thing I noticed, on a > 4G RAM machine (still waiting for the RAM to test with 8G), the disks > are not that busy. It seems to be the CPU that is the bottleneck on > FreeBSD. I tried the test with 900,000 rows instead. > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad5 > da0 in sy cs us sy id > 38 1 0 271024 3439656 6329 0 0 0 4 0 0 1703 1709 > 120725 41505 44 18 37 > 0 38 0 271024 3438860 9590 0 0 0 8 0 0 1932 1938 > 183844 47958 70 28 3 > 6 33 0 271024 3438120 7814 0 0 0 0 0 0 1823 1827 > 169969 44914 62 25 13 > 39 0 0 271024 3437960 1530 0 0 0 0 0 0 994 998 36521 > 14927 13 8 79 > 38 1 0 271024 3437244 7374 0 0 0 0 0 0 2724 2731 > 173493 51821 71 20 9 > 0 39 0 271024 3436620 4773 0 0 0 8 0 0 2727 2734 > 125699 39962 47 20 33 > 37 1 0 271024 3435836 6884 0 0 0 8 0 0 2796 2804 > 177453 58430 70 26 4 > > I am also using ULE, pgsql with WITH_THREADSAFE=true and not using > TCP to connect. How busy were your disks in your testing ? Approx. 15-25 MB with spikes at approx. 45-50 MB. Context-swithces were sometimes 140.000 with 20-30 clients (systat -vmstat 1). -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 14:36:22 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC36C16A476 for ; Tue, 29 Jan 2008 14:36:22 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) by mx1.freebsd.org (Postfix) with ESMTP id 9F0A813C468 for ; Tue, 29 Jan 2008 14:36:22 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.2/8.14.2) with ESMTP id m0TEIcAa028487; Tue, 29 Jan 2008 09:18:38 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m0TEIbsw076723 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Jan 2008 09:18:38 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200801291418.m0TEIbsw076723@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 29 Jan 2008 09:16:45 -0500 To: "Claus Guttesen" From: Mike Tancsa In-Reply-To: References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 14:36:23 -0000 At 03:46 PM 1/28/2008, Claus Guttesen wrote: >I had (allready) saved the thread in my mail-account so I could look >it up before I started testing. :-) So I compiled postgresql with the >option WITH_THREADSAFE=true and used sysbench with --pgsql-host="" . >As pointed out by Ivan my test also involved r/w whereas the thread >you (probably) mention at >http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/e224cd4f76e9ec2d >is a read-only test. > >I forgot to mention in my first post that I'm using ULE. The p800 >controller has a (factory set) 25/75 read/write cache ratio. I am still going through my own testing. One thing I noticed, on a 4G RAM machine (still waiting for the RAM to test with 8G), the disks are not that busy. It seems to be the CPU that is the bottleneck on FreeBSD. I tried the test with 900,000 rows instead. procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad5 da0 in sy cs us sy id 38 1 0 271024 3439656 6329 0 0 0 4 0 0 1703 1709 120725 41505 44 18 37 0 38 0 271024 3438860 9590 0 0 0 8 0 0 1932 1938 183844 47958 70 28 3 6 33 0 271024 3438120 7814 0 0 0 0 0 0 1823 1827 169969 44914 62 25 13 39 0 0 271024 3437960 1530 0 0 0 0 0 0 994 998 36521 14927 13 8 79 38 1 0 271024 3437244 7374 0 0 0 0 0 0 2724 2731 173493 51821 71 20 9 0 39 0 271024 3436620 4773 0 0 0 8 0 0 2727 2734 125699 39962 47 20 33 37 1 0 271024 3435836 6884 0 0 0 8 0 0 2796 2804 177453 58430 70 26 4 I am also using ULE, pgsql with WITH_THREADSAFE=true and not using TCP to connect. How busy were your disks in your testing ? ---Mike From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 17:06:00 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D7D616A419 for ; Tue, 29 Jan 2008 17:06:00 +0000 (UTC) (envelope-from amesbury@umn.edu) Received: from mta-a2.tc.umn.edu (mta-a2.tc.umn.edu [134.84.119.206]) by mx1.freebsd.org (Postfix) with ESMTP id 2A72F13C4CC for ; Tue, 29 Jan 2008 17:05:59 +0000 (UTC) (envelope-from amesbury@umn.edu) Received: from paulaner.oitsec.umn.edu (paulaner.oitsec.umn.edu [160.94.247.212]) by mta-a2.tc.umn.edu (UMN smtpd) with ESMTP for ; Tue, 29 Jan 2008 11:05:56 -0600 (CST) X-Umn-Remote-Mta: [N] paulaner.oitsec.umn.edu [160.94.247.212] #+LO+TS+AU Message-ID: <479F5CF4.4020000@umn.edu> Date: Tue, 29 Jan 2008 11:05:56 -0600 From: Alan Amesbury User-Agent: Thunderbird 2.0.0.9 (X11/20071203) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <478FBC91.1060606@umn.edu> In-Reply-To: <478FBC91.1060606@umn.edu> X-Enigmail-Version: 0.95.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: UCD-MIB for bsnmpd? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 17:06:00 -0000 Alan Amesbury wrote: [snip] > I'd like to run bsnmpd, but need the UCD-MIB for other performance > monitoring. I could run net-snmp and proxy requests through it to > bsnmpd, but that strikes me as inelegant. There's a bsnmpd-ucd module > at Google > > http://bsnmp-ucd.googlecode.com/ > > > which is seems to work, except for some minor bugs in what it reports > for laLoadFloat.[123]. Have any of you experience with this? Are there > plans to add it to the ports tree? [snip] Since no one else responded to this thread, I'm assuming I'm the only one using this feature... at least on *this* list. :-) For what it's worth, v0.1.3 appears to work as expected. I've been able to narrow down some performance problems on one of my systems. (It appears to be dropping packets when traffic exceeds ~120Kpkts/sec, vs. its near twin which seems fine at ~190Kpkts/sec). It'd still be nice to get bsnmp-ucd into the stock ports tree, though, as I think other people might eventually benefit from it. Should that request be done through a PR? As always, thanks in advance! -- Alan Amesbury University of Minnesota From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 18:09:34 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9A9016A46C for ; Tue, 29 Jan 2008 18:09:34 +0000 (UTC) (envelope-from prvs=191478baec=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2A413C469 for ; Tue, 29 Jan 2008 18:09:34 +0000 (UTC) (envelope-from prvs=191478baec=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201629338; x=1202234138; q=dns/txt; h=Received: Message-ID:From:To:Subject:Date:MIME-Version:Content-Type: Content-Transfer-Encoding; bh=NMVdzlz7wpAhjaUgtSUnXqU/GtExF7Qe7L j2gftzQLk=; b=jh0zkqrEH9I6pDZaz53G4yPwypUwN+xc0cyr0q7EgoFyaAGPoz sT4qroGrJaMobynkfWZ6se+GQvBbrPlg8eSCp/j6Y75woWjTINcyBgn10pow1KUv dkkhT2KQ/6H9MLuCcvQV7FyJvRZeUNMXIAweRdzpRBO5DoEGdimZm7pcY= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004955613.msg for ; Tue, 29 Jan 2008 17:55:38 +0000 Message-ID: <030401c862a0$23918d50$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: Date: Tue, 29 Jan 2008 17:55:30 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=191478baec=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-Spam-Processed: mail1.multiplay.co.uk, Tue, 29 Jan 2008 17:55:38 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Tue, 29 Jan 2008 17:55:38 +0000 Subject: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 18:09:34 -0000 I'm just in the midst of setting up a new machine using 7.0-PRERELEASE and while running newfs to init the data partitions the entire machine stalled for a good 20seconds when processing a 500GB partition. I had a number of windows open at the time including:- 1. gstat 2. top showing IO inc system @ 1 second intervals 3. newfs window The newfs had completed its output, gstat was showing 95% usage on da0 and da0s3g and top was showing no IO users ( all 0% ). At this point everything froze for ~ 20 seconds then returned to normal. The machine is running with ULE on 7.0 as mention using an Areca 1220 controller over 8 disks in RAID 6 + Hotspare. 17:44:38 => 17:45:05 Rerunning the newfs reproduces the issue which is good so where do I go from here? I'm really interested in looking into this as we have production machines which are also suffering from intermittent stalls which could well be related. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue Jan 29 19:45:26 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1569316A41A for ; Tue, 29 Jan 2008 19:45:26 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from mail.sweeplist.com (mail.sweeplist.com [77.93.199.51]) by mx1.freebsd.org (Postfix) with ESMTP id CDD6813C45B for ; Tue, 29 Jan 2008 19:45:25 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from [192.168.1.2] (r3a200.net.upc.cz [213.220.192.200]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sweeplist.com (Postfix) with ESMTPSA id 0C17167875; Tue, 29 Jan 2008 20:45:24 +0100 (CET) Message-ID: <479F825A.9040200@quip.cz> Date: Tue, 29 Jan 2008 20:45:30 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: cz, cs, en, en-us MIME-Version: 1.0 To: Alan Amesbury References: <478FBC91.1060606@umn.edu> <479F5CF4.4020000@umn.edu> In-Reply-To: <479F5CF4.4020000@umn.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: UCD-MIB for bsnmpd? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jan 2008 19:45:26 -0000 Alan Amesbury wrote: > Alan Amesbury wrote: > > [snip] > >>I'd like to run bsnmpd, but need the UCD-MIB for other performance >>monitoring. I could run net-snmp and proxy requests through it to >>bsnmpd, but that strikes me as inelegant. There's a bsnmpd-ucd module >>at Google >> >> http://bsnmp-ucd.googlecode.com/ >> >> >>which is seems to work, except for some minor bugs in what it reports >>for laLoadFloat.[123]. Have any of you experience with this? Are there >>plans to add it to the ports tree? > > [snip] > > Since no one else responded to this thread, I'm assuming I'm the only > one using this feature... at least on *this* list. :-) For what it's > worth, v0.1.3 appears to work as expected. I've been able to narrow > down some performance problems on one of my systems. (It appears to be > dropping packets when traffic exceeds ~120Kpkts/sec, vs. its near twin > which seems fine at ~190Kpkts/sec). > > It'd still be nice to get bsnmp-ucd into the stock ports tree, though, > as I think other people might eventually benefit from it. Should that > request be done through a PR? > > As always, thanks in advance! I am not using bsnmp-ucd so I will not help you with problems. If you want it in the ports tree, I think the best way to do it is contacting the author of bsnmp-ucd to ask him to do the port / submit it (by PR) to the official ports tree. Or you can do it your self and submit it and be the maintainer of the port. Once it hits the ports tree, it gets better attention and bugs will be (can be) fixed faster. Miroslav Lachman From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 02:28:50 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7D5316A477 for ; Wed, 30 Jan 2008 02:28:50 +0000 (UTC) (envelope-from Jinmei_Tatuya@isc.org) Received: from mon.jinmei.org (mon.jinmei.org [IPv6:2001:4f8:3:36::162]) by mx1.freebsd.org (Postfix) with ESMTP id 817B113C4F2 for ; Wed, 30 Jan 2008 02:28:45 +0000 (UTC) (envelope-from Jinmei_Tatuya@isc.org) Received: from dhcp-182.sql1.isc.org (unknown [IPv6:2001:4f8:3:bb:217:f2ff:fee0:a91f]) by mon.jinmei.org (Postfix) with ESMTP id 275CC33C59; Tue, 29 Jan 2008 18:28:45 -0800 (PST) Date: Tue, 29 Jan 2008 18:28:44 -0800 Message-ID: From: JINMEI Tatuya / =?ISO-2022-JP?B?GyRCP0BMQEMjOkgbKEI=?= To: Attila Nagy In-Reply-To: <479F02A7.9020607@fsn.hu> References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> <479F02A7.9020607@fsn.hu> User-Agent: Wanderlust/2.14.0 (Africa) Emacs/22.0 Mule/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: multipart/mixed; boundary="Multipart_Tue_Jan_29_18:28:43_2008-1" X-Mailman-Approved-At: Wed, 30 Jan 2008 03:24:50 +0000 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 02:28:50 -0000 --Multipart_Tue_Jan_29_18:28:43_2008-1 Content-Type: text/plain; charset=US-ASCII At Tue, 29 Jan 2008 11:40:39 +0100, Attila Nagy wrote: > >> Without threading I don't see this effect, the memory usage stops at a > >> sane limit and it's size can be affected by setting the max-cache-size > >> option. > >> > >> I don't think you would gain anything usable with that, am I right? > > > > Right. Can you try a simpler patch that focuses on the memory usage > > status and works with threads? If so, I'll write one and send it to > > you. > > > Of course. The machines are diskless, so writing larger log files > directly is not an easy task. (syslog is ok) Okay, please use the attached patch (applicable to 9.5.0b1, and also to 9.5.0b2 when it's published). Build it with: % STD_CDEFINES='-DLRU_DEBUG2=2' ./configure --enable-threads (or set STD_CDEFINES using setenv if you use a csh variant) The log messages shouldn't be very noisy, but if you find them too frequent, rebuild it with: % STD_CDEFINES='-DLRU_DEBUG2=1' ./configure --enable-threads Note that, if this is a thread-related bug, it may not always be reproduceable; please try several times if the problem doesn't seem to happen. (BTW: did it always occur when you first found the problem?) > ps: I have an other problem. I've recently switched from a last year > 6-STABLE to 7-STABLE and got pretty bad results on the same machine with > the same bind (9.4). > The graphs are here: > http://picasaweb.google.com/nagy.attila/20080129Fbsd6vs7Bind > > The interesting part (from when the comments are valid) starts at around > the half of the picture. You can see that on FreeBSD 6, the CPU load is > pretty much good, but on 7, both the userspace and the kernelspace > activity grows significantly. I have no idea about why this happened at the moment. Do both server handle the same level of query rate? (I'm also curious what happened in the first half of the graphs for both cases). > I've used libthr on 6, and it is the default on 7 too. bind is threaded. > I use ISC_INTERNAL_MALLOC, but the effect is the same without it. This shouldn't matter because ISC_INTERNAL_MALLOC is enabled by default as of 9.4. Thanks, --- JINMEI, Tatuya Internet Systems Consortium, Inc. --Multipart_Tue_Jan_29_18:28:43_2008-1-- From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 04:55:46 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E11C16A420 for ; Wed, 30 Jan 2008 04:55:46 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from smtp3.clear.net.nz (smtp3.clear.net.nz [203.97.33.64]) by mx1.freebsd.org (Postfix) with ESMTP id 05AA713C458 for ; Wed, 30 Jan 2008 04:55:45 +0000 (UTC) (envelope-from markir@paradise.net.nz) Received: from zmori.markir.net (121-72-64-102.dsl.telstraclear.net [121.72.64.102]) by smtp3.clear.net.nz (CLEAR Net Mail) with ESMTP id <0JVF00M4NWZ1QV10@smtp3.clear.net.nz> for freebsd-performance@freebsd.org; Wed, 30 Jan 2008 17:40:14 +1300 (NZDT) Date: Wed, 30 Jan 2008 17:40:11 +1300 From: Mark Kirkwood In-reply-to: <200801291418.m0TEIbsw076723@lava.sentex.ca> To: Mike Tancsa Message-id: <479FFFAB.2040009@paradise.net.nz> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <200801291418.m0TEIbsw076723@lava.sentex.ca> User-Agent: Thunderbird 2.0.0.9 (X11/20071203) Cc: freebsd-performance@freebsd.org, Claus Guttesen Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 04:55:46 -0000 Mike Tancsa wrote: > At 03:46 PM 1/28/2008, Claus Guttesen wrote: > >> I had (allready) saved the thread in my mail-account so I could look >> it up before I started testing. :-) So I compiled postgresql with the >> option WITH_THREADSAFE=true and used sysbench with --pgsql-host="" . >> As pointed out by Ivan my test also involved r/w whereas the thread >> you (probably) mention at >> http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/e224cd4f76e9ec2d >> >> is a read-only test. >> >> I forgot to mention in my first post that I'm using ULE. The p800 >> controller has a (factory set) 25/75 read/write cache ratio. > > > I am still going through my own testing. One thing I noticed, on a 4G > RAM machine (still waiting for the RAM to test with 8G), the disks are > not that busy. It seems to be the CPU that is the bottleneck on > FreeBSD. I tried the test with 900,000 rows instead. > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad5 da0 in sy > cs us sy id > 38 1 0 271024 3439656 6329 0 0 0 4 0 0 1703 1709 > 120725 41505 44 18 37 > 0 38 0 271024 3438860 9590 0 0 0 8 0 0 1932 1938 > 183844 47958 70 28 3 > 6 33 0 271024 3438120 7814 0 0 0 0 0 0 1823 1827 > 169969 44914 62 25 13 > 39 0 0 271024 3437960 1530 0 0 0 0 0 0 994 998 36521 > 14927 13 8 79 > 38 1 0 271024 3437244 7374 0 0 0 0 0 0 2724 2731 > 173493 51821 71 20 9 > 0 39 0 271024 3436620 4773 0 0 0 8 0 0 2727 2734 > 125699 39962 47 20 33 > 37 1 0 271024 3435836 6884 0 0 0 8 0 0 2796 2804 > 177453 58430 70 26 4 > > I am also using ULE, pgsql with WITH_THREADSAFE=true and not using TCP > to connect. How busy were your disks in your testing ? Something else it may be worth noting - currently Linux 2.6 kernels are measurably faster at reading cached file pages compared to Freebsd (6 or 7) - by a factor of about 1.5 to 2 times (see thread titled "cached file read performance"), and in this sort of oltp benchmark, you may be running into this effect frequently enough to be impacting your results. regards Mark From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 09:03:54 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 409AC16A417 for ; Wed, 30 Jan 2008 09:03:54 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id EDBF013C448 for ; Wed, 30 Jan 2008 09:03:53 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from [127.0.0.1] (fw.axelero.hu [195.228.243.120]) by people.fsn.hu (Postfix) with ESMTP id 3D7866A6FE; Wed, 30 Jan 2008 10:03:49 +0100 (CET) Message-ID: <47A03D74.8040401@fsn.hu> Date: Wed, 30 Jan 2008 10:03:48 +0100 From: Attila Nagy User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?SklOTUVJIFRhdHV5YSAvIOelnuaYjumBlOWTiQ==?= References: <475B0F3E.5070100@fsn.hu> <479DFE74.8030004@fsn.hu> <479F02A7.9020607@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, bind-users@isc.org Subject: Re: max-cache-size doesn't work with 9.5.0b1 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 09:03:54 -0000 On 2008.01.30. 3:28, JINMEI Tatuya / 神жЋйЃ”哉 wrote: > At Tue, 29 Jan 2008 11:40:39 +0100, > Attila Nagy wrote: > > >>>> Without threading I don't see this effect, the memory usage stops at a >>>> sane limit and it's size can be affected by setting the max-cache-size >>>> option. >>>> >>>> I don't think you would gain anything usable with that, am I right? >>>> >>> Right. Can you try a simpler patch that focuses on the memory usage >>> status and works with threads? If so, I'll write one and send it to >>> you. >>> >>> >> Of course. The machines are diskless, so writing larger log files >> directly is not an easy task. (syslog is ok) >> > > Okay, please use the attached patch (applicable to 9.5.0b1, and also > to 9.5.0b2 when it's published). Build it with: > % STD_CDEFINES='-DLRU_DEBUG2=2' ./configure --enable-threads > (or set STD_CDEFINES using setenv if you use a csh variant) > Will try, thanks. >> ps: I have an other problem. I've recently switched from a last year >> 6-STABLE to 7-STABLE and got pretty bad results on the same machine with >> the same bind (9.4). >> The graphs are here: >> http://picasaweb.google.com/nagy.attila/20080129Fbsd6vs7Bind >> >> The interesting part (from when the comments are valid) starts at around >> the half of the picture. You can see that on FreeBSD 6, the CPU load is >> pretty much good, but on 7, both the userspace and the kernelspace >> activity grows significantly. >> > > I have no idea about why this happened at the moment. Do both server > handle the same level of query rate? (I'm also curious what happened > in the first half of the graphs for both cases). > Exactly the same (a per packet load balancer is in front of them). Even the machines are the same. I've replaced the pictures, the previous ones included some unintended reboot-n-try stuff. > >> I've used libthr on 6, and it is the default on 7 too. bind is threaded. >> I use ISC_INTERNAL_MALLOC, but the effect is the same without it. >> > > This shouldn't matter because ISC_INTERNAL_MALLOC is enabled by > default as of 9.4. > Ouch, I didn't know this. Thanks for the clarification. From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 09:31:43 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 920A516A41B; Wed, 30 Jan 2008 09:31:43 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 329C313C457; Wed, 30 Jan 2008 09:31:42 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A043FD.9090607@FreeBSD.org> Date: Wed, 30 Jan 2008 10:31:41 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Claus Guttesen References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Ivan Voras Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 09:31:43 -0000 Claus Guttesen wrote: >>> I forgot to mention in my first post that I'm using ULE. The p800 >>> controller has a (factory set) 25/75 read/write cache ratio. >> There's maybe one additional thing: do you dual-boot Linux and FreeBSD? >> If so, you'll need to set up a separate additional partition for the >> database, instead of benchmarking it with the file systems used by the >> OS, because different areas of the drive(s) have different performance - >> you can verify this with diskinfo -t. > > I installed FreeBSD onto a boot-partition (p400i-controller) and used > the external storage (p800) as database-partition (eight 15K rpm > sas-disks in raid 1+0). Same with Ubuntu. When I re-installed FreeBSD > and ubuntu I wiped and formatted the previous partitions. Ubuntu used > ext3 which I guess is default fs. Write performance is something that we are working on, expect to hear about progress over the coming weeks/months. Kris From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 10:36:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B65A616A417 for ; Wed, 30 Jan 2008 10:36:55 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 34C5213C44B for ; Wed, 30 Jan 2008 10:36:54 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JKAJF-0001FB-6T for freebsd-performance@freebsd.org; Wed, 30 Jan 2008 10:36:49 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 30 Jan 2008 10:36:49 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 30 Jan 2008 10:36:49 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 30 Jan 2008 11:37:18 +0100 Lines: 34 Message-ID: References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig4A11114D49D2C50E78B1F06C" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <47A043FD.9090607@FreeBSD.org> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 10:36:55 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig4A11114D49D2C50E78B1F06C Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Kris Kennaway wrote: > Write performance is something that we are working on, expect to hear > about progress over the coming weeks/months. Do you have some notes or descriptions about what is being worked on? I'm currently doing some file system benchmarking for internal purposes and I'm seeing sequential read/write performance nontrivially lower on FreeBSD+UFS than on Linux+ext3. The Linux combination has ~~20% faster writes and ~~60% faster reads, on a hardware RAID10 of four 15kRPM drives, IBM ServeRAID 8 (file system ops / nonsequential are worse - send me a private message for details). Is the work that's been going on about lock contention or something at the low levels of the file system? --------------enig4A11114D49D2C50E78B1F06C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHoFNeldnAQVacBcgRAnJWAKD4FLYqJkeK64S70JeLR5KT7qhMaACg0cbv YdtnStR6Iy4wDtWUH8YP6M4= =t/Wk -----END PGP SIGNATURE----- --------------enig4A11114D49D2C50E78B1F06C-- From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 15:18:40 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AA6116A41A for ; Wed, 30 Jan 2008 15:18:40 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id DCEF813C458 for ; Wed, 30 Jan 2008 15:18:39 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JKEhl-0006hO-MR for freebsd-performance@freebsd.org; Wed, 30 Jan 2008 15:18:25 +0000 Received: from 213.202.123.79 ([213.202.123.79]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 30 Jan 2008 15:18:25 +0000 Received: from ivoras by 213.202.123.79 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 30 Jan 2008 15:18:25 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 30 Jan 2008 16:18:29 +0100 Lines: 7 Message-ID: References: <030401c862a0$23918d50$b6db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 213.202.123.79 User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) In-Reply-To: <030401c862a0$23918d50$b6db87d4@multiplay.co.uk> X-Enigmail-Version: 0.94.1.0 Sender: news Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 15:18:40 -0000 Steven Hartland wrote: > The machine is running with ULE on 7.0 as mention using an Areca 1220 > controller over 8 disks in RAID 6 + Hotspare. I'd suggest you first try to reproduce the stall without ULE, while keeping all other parameters exactly the same. From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 16:55:25 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F30716A419; Wed, 30 Jan 2008 16:55:25 +0000 (UTC) (envelope-from prvs=19159ab51a=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 6E8AD13C467; Wed, 30 Jan 2008 16:55:24 +0000 (UTC) (envelope-from prvs=19159ab51a=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201711399; x=1202316199; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=AOa1Ec9K0lbQX9YL+IP0E Z1PbDeRlAbPDGj0SfE5ESQ=; b=YJI1nvHrMzpZGD37SJQmECXPhWqVCoEZo12ME rBZemnAM+M4hgUXDVJ63Pbr3nEgryM392aFUfnK4js19gIy1uVZ0drlFe92MJiO5 vBNsyXFjR9ynMDUVfE608x8sm9Px6atouU+kK9t5tf3cR3xs+CdNOJQqVL2Zs6JZ dwc6lU= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004961055.msg; Wed, 30 Jan 2008 16:43:18 +0000 Message-ID: <025801c8635f$35377910$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: , "Ivan Voras" References: <030401c862a0$23918d50$b6db87d4@multiplay.co.uk> Date: Wed, 30 Jan 2008 16:43:14 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=19159ab51a=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Wed, 30 Jan 2008 16:43:18 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 30 Jan 2008 16:43:19 +0000 Cc: Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 16:55:25 -0000 ----- Original Message ----- From: "Ivan Voras" > Steven Hartland wrote: > >> The machine is running with ULE on 7.0 as mention using an Areca 1220 >> controller over 8 disks in RAID 6 + Hotspare. > > I'd suggest you first try to reproduce the stall without ULE, while > keeping all other parameters exactly the same. Thanks for the idea Ivan, easy one to try. Just in the process of doing a update to the latest 7.0 so will do than then try BSD if no change. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 17:13:13 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE68416A41B for ; Wed, 30 Jan 2008 17:13:13 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 3C3C313C44B for ; Wed, 30 Jan 2008 17:13:13 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id D6A511B10EE7; Wed, 30 Jan 2008 18:13:11 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_48 autolearn=no version=3.2.3 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 6D5551B10C26 for ; Wed, 30 Jan 2008 18:13:08 +0100 (CET) Message-ID: <47A0B023.5020401@moneybookers.com> Date: Wed, 30 Jan 2008 19:13:07 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (X11/20071120) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4794E6CC.1050107@moneybookers.com> In-Reply-To: <4794E6CC.1050107@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5613/Wed Jan 30 17:34:20 2008 on blah.cmotd.com X-Virus-Status: Clean Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 17:13:13 -0000 Greetings, After playing with many settings and testing various configuration, now I'm able to to receive on bridge more then 800,000 packets/s without errors, which is amazing! Unfortunately the server behind bridge can't handle more then 250,000 packets/s Please advise how I can increase those limits? Is is possible? The servers are with 82573E Gigabit Ethernet Controller (quad port) So far I tried with lagg and ng_fec, but with them I see more problems then benefits :) Tried polling with kern.polling.user_frac from 5 to 95, different HZ, but nothing helped. Stefan Lambrev wrote: > Greetings, > > I'm trying test a bridge firewall under FreeBSD 7. > > What I have as configuration is: > > Freebsd7 (web server) - bridge (FreeBSD7) - gigabit switch - flooders. > > Both FreeBSD servers are using FreeBSD 7.0-RC1 amd64 > With netperf -l 60 -p 10303 -H 10.3.3.1 I have no problems to reach > 116MB/s > with and without pf enabled. > > But what I want to test is how well will perform the firewall during > syn floods. > For this I'm using hping3 (hping-devel in ports) to generate traffic > from flooders > to the web server. > > First think, that I notice is, that hping running on linux generate > twice more traffic compared to freebsd. > So I plan to separate a server with dual bootable linux and fbsd and > to see what's the real difference. > > Second problem that I encountered is, that when running hping from > freebsd. > It exits after few seconds/minutes with this error message: > [send_ip] sendto: No buffer space available > And this happens on FreeBSD_7 and FreeBSD 6.2-p8 too amd64) > > Can I increase those buffers ? > > I'm able to generate 24MB/s SYN flood and during my test I can see > this on the bridge firewall: > netstat -w 1 -I em0 -d - external network > input (em0) output > packets errs bytes packets errs bytes colls drops > 427613 1757 25656852 233604 0 14016924 0 0 > 428089 1274 25685358 233794 0 14025174 0 0 > 427433 1167 25645998 234775 0 14088834 0 0 > 438270 2300 26296218 233384 0 14004474 0 0 > 438425 2009 26305518 233858 0 14034114 0 0 > > and from the internal network: > input (em1) output > packets errs bytes packets errs bytes colls drops > 232912 0 13974838 425796 0 25549446 0 1334 > 234487 0 14069338 423986 0 25432026 0 1631 > 233951 0 14037178 431330 0 25880286 0 3888 > 233509 0 14010658 436496 0 26191986 0 1437 > 234181 0 14050978 430291 0 25816806 0 4001 > 234144 0 14048870 430208 0 25810206 0 1621 > 234176 0 14050678 430292 0 25828926 0 3001 > > And here is top -S > > last pid: 21830; load averages: 1.01, 0.50, > 0.72 > up 3+04:59:43 20:27:49 > 84 processes: 7 running, 60 sleeping, 17 waiting > CPU states: 0.0% user, 0.0% nice, 38.2% system, 0.0% interrupt, > 61.8% idle > Mem: 17M Active, 159M Inact, 252M Wired, 120K Cache, 213M Buf, 1548M Free > Swap: 4056M Total, 4056M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 14 root 1 171 ki31 0K 16K CPU0 0 76.8H 100.00% > idle: cpu0 > 11 root 1 171 ki31 0K 16K RUN 3 76.0H 100.00% > idle: cpu3 > 25 root 1 -68 - 0K 16K CPU1 1 54:26 86.28% em0 > taskq > 26 root 1 -68 - 0K 16K CPU2 2 39:13 66.70% em1 > taskq > 12 root 1 171 ki31 0K 16K RUN 2 76.0H 37.50% idle: > cpu2 > 13 root 1 171 ki31 0K 16K RUN 1 75.9H 16.89% idle: > cpu1 > 16 root 1 -32 - 0K 16K WAIT 0 7:00 0.00% swi4: > clock sio > 51 root 1 20 - 0K 16K syncer 3 4:30 0.00% syncer > > vmstat -i > interrupt total rate > irq1: atkbd0 544 0 > irq4: sio0 10641 0 > irq14: ata0 1 0 > irq19: uhci1+ 123697 0 > cpu0: timer 553887702 1997 > irq256: em0 48227501 173 > irq257: em1 46331164 167 > cpu1: timer 553887682 1997 > cpu3: timer 553887701 1997 > cpu2: timer 553887701 1997 > Total 2310244334 8333 > > netstat -m > 594/2361/2955 mbufs in use (current/cache/total) > 592/1854/2446/204800 mbuf clusters in use (current/cache/total/max) > 592/1328 mbuf+clusters out of packet secondary zone in use > (current/cache) > 0/183/183/12800 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 1332K/5030K/6362K bytes allocated to network (current/cache/total) > > systat -ifstat > Interface Traffic Peak Total > bridge0 in 38.704 MB/s 38.704 MB/s > 185.924 GB > out 38.058 MB/s 38.058 MB/s > 189.855 GB > > em1 in 13.336 MB/s 13.402 MB/s > 51.475 GB > out 24.722 MB/s 24.722 MB/s > 137.396 GB > > em0 in 24.882 MB/s 24.882 MB/s > 138.918 GB > out 13.336 MB/s 13.403 MB/s > 45.886 GB > > Both FreeBSD servers have quad port intel network card, 2GB memory > em0@pci0:3:0:0: class=0x020000 card=0x10bc8086 chip=0x10bc8086 > rev=0x06 hdr=0x00 > vendor = 'Intel Corporation' > device = '82571EB Gigabit Ethernet Controller (Copper)' > class = network > subclass = ethernet > > Firewall server is running on CPU: Intel(R) Xeon(R) X3220 @ 2.40GHz > (quad core) > Web server is running on Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (dual core) > > So in brief how can I get rid of "No buffer space available", > increase the sent rate of hping in FreeBSD and get rid of dropped > packets on rates like 24MB/s :) > What other tests can I run (switching on of cpu cores and etc)? > Anyone interested? > > P.S. I'm using custom kernel, with SCHED_ULE, both freebsds build from > source with CPUTYPE?=core2 > and net.inet.icmp.icmplim_output=0 > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 18:19:02 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4063816A41B for ; Wed, 30 Jan 2008 18:19:02 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from mx2.itu.dk (unknown [130.226.142.29]) by mx1.freebsd.org (Postfix) with ESMTP id 0B3FC13C458 for ; Wed, 30 Jan 2008 18:19:02 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from [192.168.1.148] (stud1-15.itu.dk [130.226.140.15]) by mx2.itu.dk (Postfix) with ESMTP id B0B90F4806A for ; Wed, 30 Jan 2008 19:19:00 +0100 (CET) Message-ID: <47A0BFE7.4070708@cederstrand.dk> Date: Wed, 30 Jan 2008 19:20:23 +0100 From: Erik Cederstrand User-Agent: Thunderbird 2.0.0.6 (X11/20071022) MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> In-Reply-To: <4797A802.8060509@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 18:19:02 -0000 Kris Kennaway wrote: > Robert Watson wrote: > > One thing I am looking at is how to best create a library of world > tarballs that can be used to populate a nfsroot (or hybrid of periodic > tarballs + binary diffs to save space). Then you could provide your > benchmark in a standardized format (start/end/cleanup scripts, etc) and > tell a machine "go and run this benchmark on every daily snapshot for > the last year and give me the numbers". Actually, this just bit me. To make packages for installation, I do a standard installworld/kernel/distribution into a directory and a chroot install of the necessary ports. I create a .tgz and the end result is around 90MB (takes ca. 2 hours). Now I have 186 of those, and my file system is full (it's only 15GB, but even a 500GB disk will fill up in 2-3 months with a fast build server). I'd like a situation where I can very quickly set up a slave with a specific version of FreeBSD to run additional tests or provide shell access to a developer. This currently involves adding an entry to a queue, rebooting and waiting 2 minutes. Quick and easy, but the archiving strategy is obviously very inefficient. I'm thinking of a couple of options: 1. Having one full install per month and archiving the rest as diffs against that by recursively bsdiff'ing every file in the tree (I could bsdiff a whole tarball, but bsdiff is very memory-intensive). Quick test: 25 mins. 2. Make a hash of all files and only store the binaries where the hash is different from the monthly tarball. Faster than 1., but less effective. Quick test: 5 mins. 3. Use some kind of VCS. My experience with Subversion and binary files is that it's very slow. 4. Throw hardware at the problem. I'd say it should not take more than 10 mins to recreate an archived version. Any thoughts? Thanks, Erik From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 19:00:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0F2B16A46B for ; Wed, 30 Jan 2008 19:00:05 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 25D2213C4E7 for ; Wed, 30 Jan 2008 19:00:04 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.1/8.13.8) with ESMTP id m0UJ00nN019125; Wed, 30 Jan 2008 13:00:00 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.1/8.13.8/Submit) id m0UJ00r7019105; Wed, 30 Jan 2008 13:00:00 -0600 (CST) (envelope-from brooks) Date: Wed, 30 Jan 2008 13:00:00 -0600 From: Brooks Davis To: Erik Cederstrand Message-ID: <20080130190000.GA18333@lor.one-eyed-alien.net> References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Dxnq1zWXvFF0Q93v" Content-Disposition: inline In-Reply-To: <47A0BFE7.4070708@cederstrand.dk> User-Agent: Mutt/1.5.16 (2007-06-09) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Wed, 30 Jan 2008 13:00:00 -0600 (CST) Cc: freebsd-performance@freebsd.org Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 19:00:05 -0000 --Dxnq1zWXvFF0Q93v Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote: > Kris Kennaway wrote: >> Robert Watson wrote: >> One thing I am looking at is how to best create a library of world=20 >> tarballs that can be used to populate a nfsroot (or hybrid of periodic= =20 >> tarballs + binary diffs to save space). Then you could provide your=20 >> benchmark in a standardized format (start/end/cleanup scripts, etc) and= =20 >> tell a machine "go and run this benchmark on every daily snapshot for th= e=20 >> last year and give me the numbers". >=20 > Actually, this just bit me. To make packages for installation, I do a=20 > standard installworld/kernel/distribution into a directory and a chroot= =20 > install of the necessary ports. I create a .tgz and the end result is=20 > around 90MB (takes ca. 2 hours). Now I have 186 of those, and my file=20 > system is full (it's only 15GB, but even a 500GB disk will fill up in 2-3= =20 > months with a fast build server). >=20 > I'd like a situation where I can very quickly set up a slave with a=20 > specific version of FreeBSD to run additional tests or provide shell acce= ss=20 > to a developer. This currently involves adding an entry to a queue,=20 > rebooting and waiting 2 minutes. Quick and easy, but the archiving strate= gy=20 > is obviously very inefficient. >=20 > I'm thinking of a couple of options: > 1. Having one full install per month and archiving the rest as diffs > against that by recursively bsdiff'ing every file in the tree (I > could bsdiff a whole tarball, but bsdiff is very memory-intensive). > Quick test: 25 mins. > 2. Make a hash of all files and only store the binaries where the hash > is different from the monthly tarball. Faster than 1., but less > effective. Quick test: 5 mins. > 3. Use some kind of VCS. My experience with Subversion and binary files > is that it's very slow. > 4. Throw hardware at the problem. >=20 > I'd say it should not take more than 10 mins to recreate an archived=20 > version. Any thoughts? It seems like you should be able to combine 1 and 2 with checksums to decide if you need to run diffs. I'd think that would be quite fast. Another option would be to use ZFS snapshots. As long as you avoid replaci= ng identical files unnecessicairly, you should have pretty good effencies and little overhead. -- Brooks --Dxnq1zWXvFF0Q93v Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHoMksXY6L6fI4GtQRAuKAAJ91Oleqalwg69wLylvaVqvgD9VivgCgs0xZ AazDOiwKFZzG8OIm8kPPjbw= =hZ0V -----END PGP SIGNATURE----- --Dxnq1zWXvFF0Q93v-- From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 19:16:35 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11AE916A47C; Wed, 30 Jan 2008 19:16:35 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6609413C44B; Wed, 30 Jan 2008 19:16:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A0CD11.802@FreeBSD.org> Date: Wed, 30 Jan 2008 20:16:33 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Ivan Voras References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 19:16:35 -0000 Ivan Voras wrote: > Kris Kennaway wrote: > >> Write performance is something that we are working on, expect to hear >> about progress over the coming weeks/months. > > Do you have some notes or descriptions about what is being worked on? > I'm currently doing some file system benchmarking for internal purposes > and I'm seeing sequential read/write performance nontrivially lower on > FreeBSD+UFS than on Linux+ext3. The Linux combination has ~~20% faster > writes and ~~60% faster reads, on a hardware RAID10 of four 15kRPM > drives, IBM ServeRAID 8 (file system ops / nonsequential are worse - > send me a private message for details). Is the work that's been going on > about lock contention or something at the low levels of the file system? > Rewrite of the lockmgr primitive, for starters. Then we'll see what remains. Kris From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 19:57:31 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E490516A46E for ; Wed, 30 Jan 2008 19:57:31 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.236]) by mx1.freebsd.org (Postfix) with ESMTP id 97DE313C459 for ; Wed, 30 Jan 2008 19:57:31 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by wx-out-0506.google.com with SMTP id i29so371190wxd.7 for ; Wed, 30 Jan 2008 11:57:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=CHV20AFjpSoka8ufGlfpaOhNvslmAmpyUjjkAHerx4w=; b=XUx0pkByBKi9e7NFD1Ubu56azG/opNs4g+MqxOZqQmAfShjy8GQw2RdQqMJF3c+b7uq5tMK322d7+JXRSY01xaGKFCv4bObF+6SRNlUmG5fecaa0c+CoBEzAbvSm857WIT7UzWotr+A18hzcCMVdCqlE0at6fMXRs7O5zq4Vq7Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=F7GACIUUeUiuJMy2kzxCl0gKTxqbvDUndThnw2hyy6JkUQiLQ5y+nnUEzMN4MEjUGPJlTm+VJN38oK1wJ6O/0T6s9nDALhi+9iVHZMQRG282dk1uB3FHPGuBhL6VNF40Gmb3gmRScJFlsX7al93Dy6X4CVUDyFyqLi2yTX14mCI= Received: by 10.140.134.15 with SMTP id h15mr844638rvd.51.1201723050137; Wed, 30 Jan 2008 11:57:30 -0800 (PST) Received: by 10.141.63.14 with HTTP; Wed, 30 Jan 2008 11:57:30 -0800 (PST) Message-ID: <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> Date: Wed, 30 Jan 2008 20:57:30 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Kris Kennaway" In-Reply-To: <47A0CD11.802@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <200801281024.11571.darcyb@commandprompt.com> <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> X-Google-Sender-Auth: 0b6c3292c640f327 Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 19:57:32 -0000 On 30/01/2008, Kris Kennaway wrote: > Rewrite of the lockmgr primitive, for starters. Then we'll see what > remains. Ok, I know about the lockmgr efforts, and they will surely help some loads. I'll try to compile the results I've been talking about in a few days and post them. From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 21:44:09 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4E9616A419; Wed, 30 Jan 2008 21:44:09 +0000 (UTC) (envelope-from prvs=19159ab51a=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id A93F113C448; Wed, 30 Jan 2008 21:44:08 +0000 (UTC) (envelope-from prvs=19159ab51a=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201729345; x=1202334145; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=xMJ68fIMT5T4DMfxXEEN3 xooSDIi2H22PZLzId4ZZEU=; b=NOi5ua3SwGNPJsH63hO5pcaVWg5qCG9t7pdhQ 0JusCW6dJAC68ssY12jPLCNYBhOMd1KhxElMZD0kYYc02uArRPesAeSEk/Q0Bcw4 33bKen/K4Q+MlFIEkEaYtAElI2nZuhicXQPaMTtDvNyuIIjMpj8t7kus3/I+YTdg O7ie3s= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004962571.msg; Wed, 30 Jan 2008 21:42:24 +0000 Message-ID: <008201c86388$fd159010$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: , "Ivan Voras" References: <030401c862a0$23918d50$b6db87d4@multiplay.co.uk> Date: Wed, 30 Jan 2008 21:42:16 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=19159ab51a=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Wed, 30 Jan 2008 21:42:24 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Wed, 30 Jan 2008 21:42:25 +0000 Cc: freebsd-stable@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 21:44:09 -0000 ----- Original Message ----- From: "Ivan Voras" >> The machine is running with ULE on 7.0 as mention using an Areca 1220 >> controller over 8 disks in RAID 6 + Hotspare. > > I'd suggest you first try to reproduce the stall without ULE, while > keeping all other parameters exactly the same. Ok tried with an updated 7 world / kernel as of this afternoon and with 4BSD instead of ULE and no difference the machine still locks up with no activity for anywhere from 20 to 30 seconds. Here's a snapshot from top under cpu and io modes when the stall has occured [top] last pid: 1102; load averages: 0.02, 0.08, 0.07 up 0+00:09:37 21:39:13 162 processes: 4 running, 145 sleeping, 13 waiting CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 60M Active, 19M Inact, 54M Wired, 56K Cache, 27M Buf, 3809M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 1 171 ki31 0K 16K RUN 0 8:59 97.90% idle: cpu0 11 root 1 171 ki31 0K 16K RUN 1 8:57 95.80% idle: cpu1 1102 root 1 -8 0 4752K 1256K physrd 1 0:01 19.64% newfs 4 root 1 -8 - 0K 16K - 0 0:00 0.10% g_down 1048 root 1 96 0 7656K 2544K CPU0 0 0:01 0.00% top 1054 root 1 96 0 7656K 2348K CPU1 1 0:01 0.00% top 863 root 1 96 0 131M 15768K select 0 0:00 0.00% httpd 1055 root 1 96 0 32928K 4656K select 0 0:00 0.00% sshd last pid: 1102; load averages: 0.02, 0.08, 0.07 up 0+00:09:37 21:39:13 162 processes: 4 running, 145 sleeping, 13 waiting CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 60M Active, 19M Inact, 54M Wired, 56K Cache, 27M Buf, 3809M Free Swap: 4096M Total, 4096M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 12 root 9 154 0 0 0 0 0.00% idle: cpu0 11 root 28 5 0 0 0 0 0.00% idle: cpu1 1102 root 5 0 0 0 0 0 0.00% newfs 4 root 14 0 0 0 0 0 0.00% g_down 1048 root 1 0 0 0 0 0 0.00% top 1054 root 1 0 0 0 0 0 0.00% top 863 root 1 0 0 0 0 0 0.00% httpd [/top] ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Wed Jan 30 23:54:19 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C38816A417; Wed, 30 Jan 2008 23:54:19 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (pool-96-225-216-68.ptldor.fios.verizon.net [96.225.216.68]) by mx1.freebsd.org (Postfix) with ESMTP id 3CA8B13C457; Wed, 30 Jan 2008 23:54:19 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (localhost.home.localnet [127.0.0.1]) by schitzo.solgatos.com (8.14.1/8.13.8) with ESMTP id m0UNFMhY008551; Wed, 30 Jan 2008 15:15:22 -0800 Received: from sopwith.solgatos.com (uucp@localhost) by schitzo.solgatos.com (8.14.1/8.13.4/Submit) with UUCP id m0UNFMwN008548; Wed, 30 Jan 2008 15:15:22 -0800 Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id XAA22476; Wed, 30 Jan 2008 23:07:23 GMT Message-Id: <200801302307.XAA22476@sopwith.solgatos.com> To: "Steven Hartland" In-reply-to: Your message of "Wed, 30 Jan 2008 21:42:16 GMT." <008201c86388$fd159010$b6db87d4@multiplay.co.uk> Date: Wed, 30 Jan 2008 15:07:23 +0000 From: Dieter Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jan 2008 23:54:19 -0000 In message <008201c86388$fd159010$b6db87d4@multiplay.co.uk>, "Steven Hartland" writes: > From: "Ivan Voras" > >> The machine is running with ULE on 7.0 as mention using an Areca 1220 > >> controller over 8 disks in RAID 6 + Hotspare. > > > > I'd suggest you first try to reproduce the stall without ULE, while > > keeping all other parameters exactly the same. > > Ok tried with an updated 7 world / kernel as of this afternoon and with 4BSD > instead of ULE and no difference the machine still locks up with no activity > for anywhere from 20 to 30 seconds. > > Here's a snapshot from top under cpu and io modes when the stall has occured > [top] > last pid: 1102; load averages: 0.02, 0.08, 0.07 up 0+00:09:37 21:39:13 > 162 processes: 4 running, 145 sleeping, 13 waiting > CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle > Mem: 60M Active, 19M Inact, 54M Wired, 56K Cache, 27M Buf, 3809M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root 1 171 ki31 0K 16K RUN 0 8:59 97.90% idle: cpu0 > 11 root 1 171 ki31 0K 16K RUN 1 8:57 95.80% idle: cpu1 > 1102 root 1 -8 0 4752K 1256K physrd 1 0:01 19.64% newfs > 4 root 1 -8 - 0K 16K - 0 0:00 0.10% g_down > 1048 root 1 96 0 7656K 2544K CPU0 0 0:01 0.00% top > 1054 root 1 96 0 7656K 2348K CPU1 1 0:01 0.00% top > 863 root 1 96 0 131M 15768K select 0 0:00 0.00% httpd > 1055 root 1 96 0 32928K 4656K select 0 0:00 0.00% sshd > > > last pid: 1102; load averages: 0.02, 0.08, 0.07 up 0+00:09:37 21:39:13 > 162 processes: 4 running, 145 sleeping, 13 waiting > CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle > Mem: 60M Active, 19M Inact, 54M Wired, 56K Cache, 27M Buf, 3809M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND > 12 root 9 154 0 0 0 0 0.00% idle: cpu0 > 11 root 28 5 0 0 0 0 0.00% idle: cpu1 > 1102 root 5 0 0 0 0 0 0.00% newfs > 4 root 14 0 0 0 0 0 0.00% g_down > 1048 root 1 0 0 0 0 0 0.00% top > 1054 root 1 0 0 0 0 0 0.00% top > 863 root 1 0 0 0 0 0 0.00% httpd > [/top] What *exactly* do you mean by > machine still locks up with no activity for anywhere from 20 to 30 seconds. Is there disk activity? (e.g. activity light(s) flashing if you have them) Does top continue to update the screen during the 20-30 seconds? I'm thinking that newfs has queued up a bunch of disk i/o, and other disk i/o gets locked out, but activities that don't require any disk i/o (like top, once it is up and running) could continue. Is that what is happening? From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 01:16:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23AE816A41A; Thu, 31 Jan 2008 01:16:55 +0000 (UTC) (envelope-from prvs=1916be8aae=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 7768B13C461; Thu, 31 Jan 2008 01:16:54 +0000 (UTC) (envelope-from prvs=1916be8aae=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201741447; x=1202346247; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=PKIMFOh9AtXOFOmaijSIk rijwcc3Q8NW2G3zlbUEkVs=; b=hy0UjjbRBVuB0j70de3AuMT8mh0B2DPIAP9ZR kQLAetq50H2tD+XyJ0wDOZpNC+54ylT7XnNFC4fUGNjN+vq383ZYKyIzv1Jc4HRx 0+Gf/dEB/HF5jKUeQfXWtsTH2ynjyP3CPYCD+4i4LAUDMvy3KRh0RjSIO8jJetGi MaTNAs= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004963360.msg; Thu, 31 Jan 2008 01:04:05 +0000 Message-ID: <013201c863a5$2a91a1c0$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Dieter" References: <200801302307.XAA22476@sopwith.solgatos.com> Date: Thu, 31 Jan 2008 01:04:00 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1916be8aae=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Thu, 31 Jan 2008 01:04:06 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 31 Jan 2008 01:04:07 +0000 Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 01:16:55 -0000 ----- Original Message ----- From: "Dieter" > What *exactly* do you mean by > >> machine still locks up with no activity for anywhere from 20 to 30 seconds. > > Is there disk activity? (e.g. activity light(s) flashing if you have them) Cant tell if there is disk activity its in a DC miles away ;) > Does top continue to update the screen during the 20-30 seconds? > > I'm thinking that newfs has queued up a bunch of disk i/o, and other > disk i/o gets locked out, but activities that don't require any disk i/o > (like top, once it is up and running) could continue. Is that what is > happening? No all sessions totally freeze. I have top set to 1 second refreshes and the clock literally stops and then 20-30 seconds later continues. All sessions are via ssh not X if that's of interest. I could understand if IO to disk for other processes became slow but everything including things like top just stop which is quite odd and concerning. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 01:27:03 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28AB316A417; Thu, 31 Jan 2008 01:27:03 +0000 (UTC) (envelope-from prvs=1916be8aae=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 56AB513C448; Thu, 31 Jan 2008 01:27:02 +0000 (UTC) (envelope-from prvs=1916be8aae=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201742676; x=1202347476; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=pbMJobgd4GWPLHKLbQZRh HJARuIu6Jh7mC4IiweTQCg=; b=reH1IqLDY9OgWoJQLb3scgXMIp6oHN+lfgLt0 5dnUvKEdU9XP0fFPVUI457wIw8Y3QhLyLjvBx9IyG1OHgLM7zphLX7sRuXg5x0cZ uX0zGHxWeXFUZCyd29wJRnLn7IGfTGPXLk+JzzWrffg3+iWbC/toHm+BS6wzaEcG zoPzD4= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004963427.msg; Thu, 31 Jan 2008 01:24:35 +0000 Message-ID: <015901c863a8$064a5750$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Dieter" References: <200801302307.XAA22476@sopwith.solgatos.com> Date: Thu, 31 Jan 2008 01:24:28 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1916be8aae=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Thu, 31 Jan 2008 01:24:35 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 31 Jan 2008 01:24:36 +0000 Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: newfs + gstat locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 01:27:03 -0000 The plot thickens.... This stall is not just related to newfs you have to have gstat running as well. If I do the newfs without gstat running then no stall occurs. As soon as Im running gstat while doing the newfs then everything locks as described. Running truss on gstat shows the issue / cause I believe but I dont know what it means:- [truss -o t.txt -p 61629 -d] 9.008933817 nanosleep({1.000000000}) = 0 (0x0) 9.008969017 gettimeofday({1201742426.147393},0x0) = 0 (0x0) 9.009009804 poll({0/POLLIN},1,0) = 0 (0x0) 9.009040534 gettimeofday({1201742426.147465},0x0) = 0 (0x0) 9.009076852 clock_gettime(0,{1201742426.147501706}) = 0 (0x0) 9.009294477 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ 0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) 9.009335823 poll({0/POLLIN},1,0) = 0 (0x0) 9.009387785 poll({0/POLLIN},1,0) = 0 (0x0) 9.009457626 write(1,"\^[[4;11H 5\^[[6C2 32 467.8"...,213) = 213 (0xd5) 9.009488636 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) 10.009930312 nanosleep({1.000000000}) = 0 (0x0) 10.009963836 gettimeofday({1201742427.148388},0x0) = 0 (0x0) 10.010005182 poll({0/POLLIN},1,0) = 0 (0x0) 10.010036192 gettimeofday({1201742427.148461},0x0) = 0 (0x0) 10.010073068 clock_gettime(0,{1201742427.148497922}) = 0 (0x0) 10.010292369 mmap(0x801000000,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34376515584 (0x801000000) 10.010327569 __sysctl(0x7fffffffe6c0,0x2,0x7fffffffe650,0x7fffffffe6b8,0x800844970,0x11) = 0 (0x0) 25.052947791 __sysctl(0x7fffffffe650,0x3,0x801000000,0x7fffffffe720,0x0,0x0) = 0 (0x0) 25.054030610 munmap(0x801000000,1048576) = 0 (0x0) 25.055022356 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ 0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) 25.055067892 poll({0/POLLIN},1,0) = 0 (0x0) 25.055130470 poll({0/POLLIN},1,0) = 0 (0x0) 25.055230203 write(1,"\^[[4;11H1\^[[7C4 64 203.4"...,203) = 203 (0xcb) 25.055263448 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) 26.055866597 nanosleep({1.000000000}) = 0 (0x0) 26.055900400 gettimeofday({1201742443.194324},0x0) = 0 (0x0) 26.055940070 poll({0/POLLIN},1,0) = 0 (0x0) 26.055969962 gettimeofday({1201742443.194394},0x0) = 0 (0x0) 26.056009073 clock_gettime(0,{1201742443.194433649}) = 0 (0x0) 26.056240388 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ 0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) 26.056280896 poll({0/POLLIN},1,0) = 0 (0x0) 26.056334534 poll({0/POLLIN},1,0) = 0 (0x0) 26.056420299 poll({0/POLLIN},1,0) = 0 (0x0) 26.056485112 write(1,"\^[[1;6H6.046s w: 1.000s\^[[4;5"...,305) = 305 (0x131) 26.056516121 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) 27.056863372 nanosleep({1.000000000}) = 0 (0x0) [/truss -o t.txt -p 61629 -d] ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 01:49:18 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D96A116A418; Thu, 31 Jan 2008 01:49:18 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (pool-96-225-216-68.ptldor.fios.verizon.net [96.225.216.68]) by mx1.freebsd.org (Postfix) with ESMTP id 9DBDC13C442; Thu, 31 Jan 2008 01:49:18 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (localhost.home.localnet [127.0.0.1]) by schitzo.solgatos.com (8.14.1/8.13.8) with ESMTP id m0V1nIG7009607; Wed, 30 Jan 2008 17:49:18 -0800 Received: from sopwith.solgatos.com (uucp@localhost) by schitzo.solgatos.com (8.14.1/8.13.4/Submit) with UUCP id m0V1nIcR009603; Wed, 30 Jan 2008 17:49:18 -0800 Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id BAA04522; Thu, 31 Jan 2008 01:47:04 GMT Message-Id: <200801310147.BAA04522@sopwith.solgatos.com> To: "Steven Hartland" In-reply-to: Your message of "Thu, 31 Jan 2008 01:04:00 GMT." <013201c863a5$2a91a1c0$b6db87d4@multiplay.co.uk> Date: Wed, 30 Jan 2008 17:47:04 +0000 From: Dieter Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 01:49:19 -0000 > > What *exactly* do you mean by > > > >> machine still locks up with no activity for anywhere from 20 to 30 seconds. > > > > Is there disk activity? (e.g. activity light(s) flashing if you have them) > > Cant tell if there is disk activity its in a DC miles away ;) > > > Does top continue to update the screen during the 20-30 seconds? > > > > I'm thinking that newfs has queued up a bunch of disk i/o, and other > > disk i/o gets locked out, but activities that don't require any disk i/o > > (like top, once it is up and running) could continue. Is that what is > > happening? > > No all sessions totally freeze. I have top set to 1 second refreshes and > the clock literally stops and then 20-30 seconds later continues. Wow! > All sessions are via ssh not X if that's of interest. > > I could understand if IO to disk for other processes became slow but > everything including things like top just stop which is quite odd and > concerning. Ok, new idea. Your disk activity is via the network, and your sessions are via the network. Could it be that something network related is saturated? From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 13:30:58 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 924E216A420 for ; Thu, 31 Jan 2008 13:30:58 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 4227A13C4D1 for ; Thu, 31 Jan 2008 13:30:58 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A554D0.dip.t-dialin.net [84.165.84.208]) by redbull.bpaserver.net (Postfix) with ESMTP id CCF692E264; Thu, 31 Jan 2008 14:30:47 +0100 (CET) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 56D0946662; Thu, 31 Jan 2008 14:30:41 +0100 (CET) Received: (from www@localhost) by webmail.leidinger.net (8.14.2/8.13.8/Submit) id m0VDUfKV051191; Thu, 31 Jan 2008 14:30:41 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Thu, 31 Jan 2008 14:30:41 +0100 Message-ID: <20080131143041.pzb4k77xtgkw8oc0@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Thu, 31 Jan 2008 14:30:41 +0100 From: Alexander Leidinger To: Miroslav Lachman <000.fbsd@quip.cz> References: <478FBC91.1060606@umn.edu> <479F5CF4.4020000@umn.edu> <479F825A.9040200@quip.cz> In-Reply-To: <479F825A.9040200@quip.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.5) / FreeBSD-8.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.4, required 6, BAYES_00 -15.00, RDNS_DYNAMIC 0.10, SMILEY -0.50, VOWEL_URI_5 1.00) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: freebsd-performance@freebsd.org, Alan Amesbury Subject: Re: UCD-MIB for bsnmpd? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 13:30:58 -0000 Quoting Miroslav Lachman <000.fbsd@quip.cz> (from Tue, 29 Jan 2008 =20 20:45:30 +0100): > Alan Amesbury wrote: > >> Alan Amesbury wrote: >> >> [snip] >> >>> I'd like to run bsnmpd, but need the UCD-MIB for other performance >>> monitoring. I could run net-snmp and proxy requests through it to >>> bsnmpd, but that strikes me as inelegant. There's a bsnmpd-ucd module >>> at Google >>> >>> =09http://bsnmp-ucd.googlecode.com/ >>> >>> >>> which is seems to work, except for some minor bugs in what it reports >>> for laLoadFloat.[123]. Have any of you experience with this? Are there >>> plans to add it to the ports tree? >> >> [snip] >> >> Since no one else responded to this thread, I'm assuming I'm the only >> one using this feature... at least on *this* list. :-) For what it's >> worth, v0.1.3 appears to work as expected. I've been able to narrow >> down some performance problems on one of my systems. (It appears to be >> dropping packets when traffic exceeds ~120Kpkts/sec, vs. its near twin >> which seems fine at ~190Kpkts/sec). >> >> It'd still be nice to get bsnmp-ucd into the stock ports tree, though, >> as I think other people might eventually benefit from it. Should that >> request be done through a PR? >> >> As always, thanks in advance! > > I am not using bsnmp-ucd so I will not help you with problems. If you > want it in the ports tree, I think the best way to do it is contacting > the author of bsnmp-ucd to ask him to do the port / submit it (by PR) > to the official ports tree. Or you can do it your self and submit it > and be the maintainer of the port. > Once it hits the ports tree, it gets better attention and bugs will be > (can be) fixed faster. People with a strong interest in seeing this in the base system could =20 also contact syrinx@ and/or harti@, as they take care about bsnmpd. Bye, Alexander. --=20 We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. =09=09-- Larry Wall http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D 72077137 From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 16:55:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F58A16A418 for ; Thu, 31 Jan 2008 16:55:48 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.184]) by mx1.freebsd.org (Postfix) with ESMTP id C5B8113C442 for ; Thu, 31 Jan 2008 16:55:47 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so652205rvb.43 for ; Thu, 31 Jan 2008 08:55:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=cjxN3attjNZa1Rc5Xc4OUIXA0MUYGDu60sIQgugTZIg=; b=iJ7RV/UG2B4zaPBxhDByYm8NCrcHoqqByNwi1mpYxSSJtQY9O3SuNUQoFM00780PiEWsMwcw0ywYep91/ZB7aTFWKH/Z4dyqJT1QnRQ4PDHh7f7MC8Wy9KkHBHW3VoE1DU4cmLws1cb+T1N7V1Vjzui0kOS5fjf3TDmm0TqYPkA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=spl/pyJtCFiVpGA6doyRhD/xS43lt5aepE3VXs+cmfjNDAkwLUcKjnuD14ctdWuNFFRTDRmoV8PYzsK+4IuHbOXyQmk16ZuqffoYaVAgJsd9cpi44blrowx2m8p0TvftTuTF7dNygg5rASwpIBSA3ni6f1SupsPhBoUAHGym1pw= Received: by 10.141.170.10 with SMTP id x10mr1584137rvo.139.1201796795403; Thu, 31 Jan 2008 08:26:35 -0800 (PST) Received: by 10.141.170.18 with HTTP; Thu, 31 Jan 2008 08:26:35 -0800 (PST) Message-ID: <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> Date: Thu, 31 Jan 2008 18:26:35 +0200 From: "Niki Denev" Sender: ndenev@gmail.com To: freebsd-performance@freebsd.org In-Reply-To: <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> X-Google-Sender-Auth: b9288bb34249e843 X-Mailman-Approved-At: Thu, 31 Jan 2008 17:13:47 +0000 Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 16:55:59 -0000 On Jan 30, 2008 9:57 PM, Ivan Voras wrote: > On 30/01/2008, Kris Kennaway wrote: > > > Rewrite of the lockmgr primitive, for starters. Then we'll see what > > remains. > > Ok, I know about the lockmgr efforts, and they will surely help some > loads. I'll try to compile the results I've been talking about in a > few days and post them. > Just a thought on the effect that HZ has on filesystem (and overall) performance : Linux has sort of backtracked from defaulting to HZ=1000 and enable it only on kernels compiled for "Desktop" work, and setting HZ=250 for the "Server" profile. A very quick /usr/ports/benchmarks/postmark benchmark on my machine (FreeBSD 7.0 PREREL with ULE on dualcore C2D) shows this : HZ=1000 Time: 239 seconds total 122 seconds of transactions (4 per second) Files: 749 created (3 per second) Creation alone: 500 files (4 per second) Mixed with transactions: 249 files (2 per second) 267 read (2 per second) 233 appended (1 per second) 749 deleted (3 per second) Deletion alone: 498 files (498 per second) Mixed with transactions: 251 files (2 per second) Data: 573.06 megabytes read (2.40 megabytes per second) 1699.49 megabytes written (7.11 megabytes per second) HZ=250 Time: 178 seconds total 79 seconds of transactions (6 per second) Files: 749 created (4 per second) Creation alone: 500 files (5 per second) Mixed with transactions: 249 files (3 per second) 267 read (3 per second) 233 appended (2 per second) 749 deleted (4 per second) Deletion alone: 498 files (498 per second) Mixed with transactions: 251 files (3 per second) Data: 573.06 megabytes read (3.22 megabytes per second) 1699.49 megabytes written (9.55 megabytes per second) A measurable increase in performance when using HZ=250, at least in this simple benchmark. The question is do we need such sort of profiles as in Linux, i.e. HZ=250 for servers and HZ=1000 for desktops. Of course anyone can set HZ to whatever value he likes, but maybe it should be better if it is documented with some advisory settings for different workloads (server/desktop). What do you think? P.S.: Maybe it would be interesting to see how FreeBSD compares to Linux with equal HZ settings on the benchmark in the first post? From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 19:58:19 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0189316A41A; Thu, 31 Jan 2008 19:58:18 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost2.sentex.ca (smarthost2.sentex.ca [205.211.164.50]) by mx1.freebsd.org (Postfix) with ESMTP id 9D22913C43E; Thu, 31 Jan 2008 19:58:17 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost2.sentex.ca (8.14.2/8.14.2) with ESMTP id m0VJwGAn039689; Thu, 31 Jan 2008 14:58:17 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m0VJwF0f090920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 31 Jan 2008 14:58:16 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200801311958.m0VJwF0f090920@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Thu, 31 Jan 2008 15:00:20 -0500 To: "Steven Hartland" From: Mike Tancsa In-Reply-To: <015901c863a8$064a5750$b6db87d4@multiplay.co.uk> References: <200801302307.XAA22476@sopwith.solgatos.com> <015901c863a8$064a5750$b6db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org Subject: Re: newfs + gstat locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 19:58:19 -0000 At 08:24 PM 1/30/2008, Steven Hartland wrote: >The plot thickens.... This stall is not just related to newfs you have to >have gstat running as well. If I do the newfs without gstat running then >no stall occurs. As soon as Im running gstat while doing the newfs then >everything locks as described. Strange, I see the same thing sometimes. While running while true do date sleep .5 done Thu Jan 31 14:55:42 EST 2008 Thu Jan 31 14:55:42 EST 2008 Thu Jan 31 14:55:43 EST 2008 Thu Jan 31 14:55:43 EST 2008 Thu Jan 31 14:55:44 EST 2008 Thu Jan 31 14:55:44 EST 2008 Thu Jan 31 14:55:50 EST 2008 Thu Jan 31 14:55:50 EST 2008 Thu Jan 31 14:55:51 EST 2008 Thu Jan 31 14:55:51 EST 2008 Thu Jan 31 14:55:52 EST 2008 Thu Jan 31 14:55:52 EST 2008 Thu Jan 31 14:55:53 EST 2008 Thu Jan 31 14:55:53 EST 2008 Thu Jan 31 14:55:54 EST 2008 You can see it from 44 to 50 seconds, This is AMD64 da0 at arcmsr0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: 305175MB (624999424 512 byte sectors: 255H 63S/T 38904C) arcmsr0: mem 0xe8600000-0xe8600fff,0xe8000000-0xe83fffff irq 18 at device 14.0 on pci2 ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 arcmsr0: [ITHREAD] >Running truss on gstat shows the issue / cause I believe but I dont >know what it means:- >[truss -o t.txt -p 61629 -d] >9.008933817 nanosleep({1.000000000}) = 0 (0x0) >9.008969017 gettimeofday({1201742426.147393},0x0) = 0 (0x0) >9.009009804 poll({0/POLLIN},1,0) = 0 (0x0) >9.009040534 gettimeofday({1201742426.147465},0x0) = 0 (0x0) >9.009076852 clock_gettime(0,{1201742426.147501706}) = 0 (0x0) >9.009294477 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ >0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) >9.009335823 poll({0/POLLIN},1,0) = 0 (0x0) >9.009387785 poll({0/POLLIN},1,0) = 0 (0x0) >9.009457626 write(1,"\^[[4;11H 5\^[[6C2 32 467.8"...,213) = 213 (0xd5) >9.009488636 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) >10.009930312 nanosleep({1.000000000}) = 0 (0x0) >10.009963836 gettimeofday({1201742427.148388},0x0) = 0 (0x0) >10.010005182 poll({0/POLLIN},1,0) = 0 (0x0) >10.010036192 gettimeofday({1201742427.148461},0x0) = 0 (0x0) >10.010073068 clock_gettime(0,{1201742427.148497922}) = 0 (0x0) >10.010292369 >mmap(0x801000000,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) >= 34376515584 (0x801000000) >10.010327569 >__sysctl(0x7fffffffe6c0,0x2,0x7fffffffe650,0x7fffffffe6b8,0x800844970,0x11) >= 0 (0x0) >25.052947791 >__sysctl(0x7fffffffe650,0x3,0x801000000,0x7fffffffe720,0x0,0x0) = 0 (0x0) >25.054030610 munmap(0x801000000,1048576) = 0 (0x0) >25.055022356 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ >0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) >25.055067892 poll({0/POLLIN},1,0) = 0 (0x0) >25.055130470 poll({0/POLLIN},1,0) = 0 (0x0) >25.055230203 write(1,"\^[[4;11H1\^[[7C4 64 203.4"...,203) = 203 (0xcb) >25.055263448 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) >26.055866597 nanosleep({1.000000000}) = 0 (0x0) >26.055900400 gettimeofday({1201742443.194324},0x0) = 0 (0x0) >26.055940070 poll({0/POLLIN},1,0) = 0 (0x0) >26.055969962 gettimeofday({1201742443.194394},0x0) = 0 (0x0) >26.056009073 clock_gettime(0,{1201742443.194433649}) = 0 (0x0) >26.056240388 sigaction(SIGTSTP,{ SIG_IGN SA_RESTART ss_t },{ >0x800cb2470 SA_RESTART ss_t }) = 0 (0x0) >26.056280896 poll({0/POLLIN},1,0) = 0 (0x0) >26.056334534 poll({0/POLLIN},1,0) = 0 (0x0) >26.056420299 poll({0/POLLIN},1,0) = 0 (0x0) >26.056485112 write(1,"\^[[1;6H6.046s w: 1.000s\^[[4;5"...,305) = 305 (0x131) >26.056516121 sigaction(SIGTSTP,{ 0x800cb2470 SA_RESTART ss_t },0x0) = 0 (0x0) >27.056863372 nanosleep({1.000000000}) = 0 (0x0) >[/truss -o t.txt -p 61629 -d] > >================================================ >This e.mail is private and confidential between Multiplay (UK) Ltd. >and the person or entity to whom it is addressed. In the event of >misdirection, the recipient is prohibited from using, copying, >printing or otherwise disseminating it or any information contained in it. >In the event of misdirection, illegible or incomplete transmission >please telephone +44 845 868 1337 >or return the E.mail to postmaster@multiplay.co.uk. > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 20:16:30 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2FDF16A41A for ; Thu, 31 Jan 2008 20:16:30 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 2F98D13C478 for ; Thu, 31 Jan 2008 20:16:29 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JKfph-0006Vw-GH for freebsd-performance@freebsd.org; Thu, 31 Jan 2008 20:16:25 +0000 Received: from 78-1-69-12.adsl.net.t-com.hr ([78.1.69.12]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 31 Jan 2008 20:16:25 +0000 Received: from ivoras by 78-1-69-12.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 31 Jan 2008 20:16:25 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Thu, 31 Jan 2008 21:16:18 +0100 Lines: 43 Message-ID: References: <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig658431FD5296B48F8B5B4F38" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 78-1-69-12.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) In-Reply-To: <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> X-Enigmail-Version: 0.95.6 Sender: news Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 20:16:30 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig658431FD5296B48F8B5B4F38 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Niki Denev wrote: > HZ=3D1000 > Time: > 239 seconds total > 122 seconds of transactions (4 per second) > What do you think? This is a very low result :) I don't know your machine or the parameters = you used with postmark but even FreeBSD on two striped 7.5 kRPM drives=20 can achieve ~~ 110 tps (so ~~ 50 for a single drive). In the same=20 circumstances and with the right file system - reiserfs in this case -=20 Linux can achieve close to 900 tps (170 tps with ext3). This is the kind = of difference I am talking about. A change of 4 tps to 6 tps is not=20 generally useful. --------------enig658431FD5296B48F8B5B4F38 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHoiySldnAQVacBcgRAtGHAKDKx7r8o2ahpEFlYpkkpyPGnhKorgCgtq9A JJU4b0DO9R6LB5z3yEdpwCU= =JJd6 -----END PGP SIGNATURE----- --------------enig658431FD5296B48F8B5B4F38-- From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 20:42:09 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3ED1F16A468 for ; Thu, 31 Jan 2008 20:42:09 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.190]) by mx1.freebsd.org (Postfix) with ESMTP id 15CD013C4F9 for ; Thu, 31 Jan 2008 20:42:08 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so697779rvb.43 for ; Thu, 31 Jan 2008 12:42:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=bn1MGkRQqN//fO8GvLPaRy6uMPGGthZqh5xbl3q1wMs=; b=Mxvs/biBApLnSlPjtDbWQEMMIdhrfONVvNT7MzxvM+L/zwi69fksUMc8TecbcdaO/+X+o83aaki412V0Fia6h/EoeJqQsuTk1SC68JHMQOszPQoCpUJju2V6101RN5Ec7fxhzZHphWesRid/XkVjD3Dh9eeL29KI+rngFq+yS1Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=IG9QOmWFqc4ZwsTAcJ/Op5UD+XxLeWXVOMQz6EjbFLhXI/6deOAxw2EtVMWevc2u9N+k0KkNiE594phQVKa/c60bPvZVt73qAHk5a1X9ITTX3nPprZ1rDj/S8wmLs6hrL+nOXMWTe8V8xyIcUmOe23jmzIpNFMKWVcVeOa+wrAk= Received: by 10.140.207.3 with SMTP id e3mr1840129rvg.194.1201812126094; Thu, 31 Jan 2008 12:42:06 -0800 (PST) Received: by 10.141.170.18 with HTTP; Thu, 31 Jan 2008 12:42:06 -0800 (PST) Message-ID: <2e77fc10801311242n12c5b2bcseaa2b717f95c21b5@mail.gmail.com> Date: Thu, 31 Jan 2008 22:42:06 +0200 From: "Niki Denev" Sender: ndenev@gmail.com To: "Ivan Voras" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> X-Google-Sender-Auth: 424608be102b7833 X-Mailman-Approved-At: Thu, 31 Jan 2008 21:31:07 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 20:42:09 -0000 On Jan 31, 2008 10:16 PM, Ivan Voras wrote: > Niki Denev wrote: > > > HZ=1000 > > Time: > > 239 seconds total > > 122 seconds of transactions (4 per second) > > > What do you think? > > This is a very low result :) I don't know your machine or the parameters > you used with postmark but even FreeBSD on two striped 7.5 kRPM drives > can achieve ~~ 110 tps (so ~~ 50 for a single drive). In the same > circumstances and with the right file system - reiserfs in this case - > Linux can achieve close to 900 tps (170 tps with ext3). This is the kind > of difference I am talking about. A change of 4 tps to 6 tps is not > generally useful. > > What settings are you using for postmark? I ran postmark with default settings, and only changed "size" to "4096 4096000", so it could create files varying in size from 4K to 3.9M The machine is dualcore core2duo and has two 10Krpm scsi disks in geom_mirror and geom_journal on top of them. I guess that this may be because of geom_journal performance penalty, having to write everything two times. Ubench too shows no significant difference, although its not using disk subsystem at all. HZ=250 Ubench CPU: 322604 Ubench MEM: 152835 -------------------- Ubench AVG: 237719 HZ=1000 Ubench CPU: 321800 Ubench MEM: 152807 -------------------- Ubench AVG: 237303 Maybe i'll try also "bonnie++" From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 22:03:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6BE3016A417 for ; Thu, 31 Jan 2008 22:03:16 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.181]) by mx1.freebsd.org (Postfix) with ESMTP id 32EBA13C45A for ; Thu, 31 Jan 2008 22:03:16 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by py-out-1112.google.com with SMTP id u52so1200882pyb.10 for ; Thu, 31 Jan 2008 14:03:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=TixOIHh6ND+J7JNlVyrbFJbN0HsXzLNDdVq2w3Lam3k=; b=qNN4VkfvNW0iMxiAMzZtZOgCoR48OZZWSdwiIbyThvIYdfI+6r0F6H3HP43HjHoS+YTYOSl92u2wgd9N3/Qll/oh7OKV1R1yZ7S1NCLH1p9TmL0aZ34n/TqB1vJHOHGlyv74Sb1L3G6WlpBBZHmAb7PXhoTr6VeFK0bWqaiiWrI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=q+HnN1n7yK2SfBuIuss9+gF1E4MZcd7hLG/Di3GL+iQoz3gyMU71fAo+h09O/WMjxzSM6FJopsvBBNm0whW4TTMS0DcqyOCHLy3Q3s5L5l9ncLxLPzvByC8BC181yjXHgNyu8HRxGifU9e+kSdz42LcFTsS4RoJyqMNukXRbVZg= Received: by 10.140.163.3 with SMTP id l3mr1924266rve.22.1201816994495; Thu, 31 Jan 2008 14:03:14 -0800 (PST) Received: by 10.141.63.14 with HTTP; Thu, 31 Jan 2008 14:03:14 -0800 (PST) Message-ID: <9bbcef730801311403r45c54036v36b97450d0c37a65@mail.gmail.com> Date: Thu, 31 Jan 2008 23:03:14 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Niki Denev" In-Reply-To: <2e77fc10801311242n12c5b2bcseaa2b717f95c21b5@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> <2e77fc10801311242n12c5b2bcseaa2b717f95c21b5@mail.gmail.com> X-Google-Sender-Auth: 69e8ff22a18fee38 Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 22:03:16 -0000 On 31/01/2008, Niki Denev wrote: > On Jan 31, 2008 10:16 PM, Ivan Voras wrote: > > Niki Denev wrote: > > > > > HZ=1000 > > > Time: > > > 239 seconds total > > > 122 seconds of transactions (4 per second) > > > > > What do you think? > > > > This is a very low result :) I don't know your machine or the parameters > > you used with postmark but even FreeBSD on two striped 7.5 kRPM drives > > can achieve ~~ 110 tps (so ~~ 50 for a single drive). In the same > > circumstances and with the right file system - reiserfs in this case - > > Linux can achieve close to 900 tps (170 tps with ext3). This is the kind > > of difference I am talking about. A change of 4 tps to 6 tps is not > > generally useful. > > > > > > What settings are you using for postmark? > I ran postmark with default settings, and only changed "size" to "4096 > 4096000", so it could create > files varying in size from 4K to 3.9M Ok, under these circumstances+gjournal it's maybe possible to get numbers that low. I am testing operations on small files so my whole configuration is: set size 4096 65536 set read 8192 set write 8192 set number 10000 set transactions 40000 set subdirectories 1000 From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 23:30:12 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BF5F16A418; Thu, 31 Jan 2008 23:30:12 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2CDE913C465; Thu, 31 Jan 2008 23:30:10 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A25A01.5080508@FreeBSD.org> Date: Fri, 01 Feb 2008 00:30:09 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> In-Reply-To: <47A25412.3010301@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 23:30:12 -0000 Alexander Motin wrote: > Hi. > > While profiling netgraph operation on UP HEAD router I have found that > huge amount of time it spent on memory allocation/deallocation: > > 0.14 0.05 132119/545292 ip_forward [12] > 0.14 0.05 133127/545292 fxp_add_rfabuf [18] > 0.27 0.10 266236/545292 ng_package_data [17] > [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] > 0.17 0.00 545292/1733401 critical_exit [98] > 0.01 0.00 275941/679675 generic_bzero [68] > 0.01 0.00 133127/133127 mb_ctor_pack [103] > > 0.15 0.06 133100/545266 mb_free_ext [22] > 0.15 0.06 133121/545266 m_freem [15] > 0.29 0.11 266236/545266 ng_free_item [16] > [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] > 0.17 0.00 545266/1733401 critical_exit [98] > 0.00 0.04 133100/133100 mb_dtor_pack [57] > 0.00 0.00 134121/134121 mb_dtor_mbuf [111] > > I have already optimized all possible allocation calls and those that > left are practically unavoidable. But even after this kgmon tells that > 30% of CPU time consumed by memory management. > > So I have some questions: > 1) Is it real situation or just profiler mistake? > 2) If it is real then why UMA is so slow? I have tried to replace it in > some places with preallocated TAILQ of required memory blocks protected > by mutex and according to profiler I have got _much_ better results. > Will it be a good practice to replace relatively small UMA zones with > preallocated queue to avoid part of UMA calls? > 3) I have seen that UMA does some kind of CPU cache affinity, but does > it cost so much that it costs 30% CPU time on UP router? Make sure you have INVARIANTS disabled, it has a high performance cost in UMA. Kris From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 23:31:09 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92B6116A469; Thu, 31 Jan 2008 23:31:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 35CA413C45D; Thu, 31 Jan 2008 23:31:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A25A3A.80700@FreeBSD.org> Date: Fri, 01 Feb 2008 00:31:06 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> <47A255BB.3020708@FreeBSD.org> In-Reply-To: <47A255BB.3020708@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 23:31:09 -0000 Alexander Motin wrote: > Alexander Motin пишет: >> While profiling netgraph operation on UP HEAD router I have found that >> huge amount of time it spent on memory allocation/deallocation: > > I have forgotten to tell that it was mostly GENERIC kernel just built > without INVARIANTS, WITNESS and SMP but with 'profile 2'. > What is 'profile 2'? Kris From owner-freebsd-performance@FreeBSD.ORG Thu Jan 31 23:42:42 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D61A716A420 for ; Thu, 31 Jan 2008 23:42:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outC.internet-mail-service.net (outC.internet-mail-service.net [216.240.47.226]) by mx1.freebsd.org (Postfix) with ESMTP id C120913C4D9 for ; Thu, 31 Jan 2008 23:42:42 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Thu, 31 Jan 2008 15:30:24 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id B747712708F; Thu, 31 Jan 2008 15:30:22 -0800 (PST) Message-ID: <47A25A0D.2080508@elischer.org> Date: Thu, 31 Jan 2008 15:30:21 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> In-Reply-To: <47A25412.3010301@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 23:42:42 -0000 Alexander Motin wrote: > Hi. > > While profiling netgraph operation on UP HEAD router I have found that > huge amount of time it spent on memory allocation/deallocation: > > 0.14 0.05 132119/545292 ip_forward [12] > 0.14 0.05 133127/545292 fxp_add_rfabuf [18] > 0.27 0.10 266236/545292 ng_package_data [17] > [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] > 0.17 0.00 545292/1733401 critical_exit [98] > 0.01 0.00 275941/679675 generic_bzero [68] > 0.01 0.00 133127/133127 mb_ctor_pack [103] > > 0.15 0.06 133100/545266 mb_free_ext [22] > 0.15 0.06 133121/545266 m_freem [15] > 0.29 0.11 266236/545266 ng_free_item [16] > [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] > 0.17 0.00 545266/1733401 critical_exit [98] > 0.00 0.04 133100/133100 mb_dtor_pack [57] > 0.00 0.00 134121/134121 mb_dtor_mbuf [111] > > I have already optimized all possible allocation calls and those that > left are practically unavoidable. But even after this kgmon tells that > 30% of CPU time consumed by memory management. > > So I have some questions: > 1) Is it real situation or just profiler mistake? > 2) If it is real then why UMA is so slow? I have tried to replace it in > some places with preallocated TAILQ of required memory blocks protected > by mutex and according to profiler I have got _much_ better results. > Will it be a good practice to replace relatively small UMA zones with > preallocated queue to avoid part of UMA calls? > 3) I have seen that UMA does some kind of CPU cache affinity, but does > it cost so much that it costs 30% CPU time on UP router? given this information, I would add an 'item cache' in ng_base.c (hmm do I already have one?) > > Thanks! > From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 00:05:00 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4640F16A46B for ; Fri, 1 Feb 2008 00:05:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id B4E9313C4D1 for ; Fri, 1 Feb 2008 00:04:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71393496; Fri, 01 Feb 2008 01:04:57 +0200 Message-ID: <47A25412.3010301@FreeBSD.org> Date: Fri, 01 Feb 2008 01:04:50 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 01 Feb 2008 00:59:35 +0000 Cc: Subject: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 00:05:00 -0000 Hi. While profiling netgraph operation on UP HEAD router I have found that huge amount of time it spent on memory allocation/deallocation: 0.14 0.05 132119/545292 ip_forward [12] 0.14 0.05 133127/545292 fxp_add_rfabuf [18] 0.27 0.10 266236/545292 ng_package_data [17] [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] 0.17 0.00 545292/1733401 critical_exit [98] 0.01 0.00 275941/679675 generic_bzero [68] 0.01 0.00 133127/133127 mb_ctor_pack [103] 0.15 0.06 133100/545266 mb_free_ext [22] 0.15 0.06 133121/545266 m_freem [15] 0.29 0.11 266236/545266 ng_free_item [16] [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] 0.17 0.00 545266/1733401 critical_exit [98] 0.00 0.04 133100/133100 mb_dtor_pack [57] 0.00 0.00 134121/134121 mb_dtor_mbuf [111] I have already optimized all possible allocation calls and those that left are practically unavoidable. But even after this kgmon tells that 30% of CPU time consumed by memory management. So I have some questions: 1) Is it real situation or just profiler mistake? 2) If it is real then why UMA is so slow? I have tried to replace it in some places with preallocated TAILQ of required memory blocks protected by mutex and according to profiler I have got _much_ better results. Will it be a good practice to replace relatively small UMA zones with preallocated queue to avoid part of UMA calls? 3) I have seen that UMA does some kind of CPU cache affinity, but does it cost so much that it costs 30% CPU time on UP router? Thanks! -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 00:12:04 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FA6C16A418 for ; Fri, 1 Feb 2008 00:12:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id C859A13C467 for ; Fri, 1 Feb 2008 00:12:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71395333; Fri, 01 Feb 2008 01:12:01 +0200 Message-ID: <47A255BB.3020708@FreeBSD.org> Date: Fri, 01 Feb 2008 01:11:55 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org, freebsd-hackers@freebsd.org References: <47A25412.3010301@FreeBSD.org> In-Reply-To: <47A25412.3010301@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 01 Feb 2008 01:01:14 +0000 Cc: Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 00:12:04 -0000 Alexander Motin пишет: > While profiling netgraph operation on UP HEAD router I have found that > huge amount of time it spent on memory allocation/deallocation: I have forgotten to tell that it was mostly GENERIC kernel just built without INVARIANTS, WITNESS and SMP but with 'profile 2'. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 05:06:49 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AB1A16B72B for ; Fri, 1 Feb 2008 05:06:49 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id BCB7413C4D1 for ; Fri, 1 Feb 2008 05:06:48 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.local (r74-193-81-203.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.81.203]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id m114saAY013489; Thu, 31 Jan 2008 22:54:36 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <47A2A606.9080702@freebsd.org> Date: Thu, 31 Jan 2008 22:54:30 -0600 From: Eric Anderson User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Dieter References: <200801310147.BAA04522@sopwith.solgatos.com> In-Reply-To: <200801310147.BAA04522@sopwith.solgatos.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-performance@freebsd.org, Steven Hartland Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 05:06:49 -0000 Dieter wrote: >>> What *exactly* do you mean by >>> >>>> machine still locks up with no activity for anywhere from 20 to 30 seconds. >>> Is there disk activity? (e.g. activity light(s) flashing if you have them) >> Cant tell if there is disk activity its in a DC miles away ;) >> >>> Does top continue to update the screen during the 20-30 seconds? >>> >>> I'm thinking that newfs has queued up a bunch of disk i/o, and other >>> disk i/o gets locked out, but activities that don't require any disk i/o >>> (like top, once it is up and running) could continue. Is that what is >>> happening? >> No all sessions totally freeze. I have top set to 1 second refreshes and >> the clock literally stops and then 20-30 seconds later continues. > > Wow! > >> All sessions are via ssh not X if that's of interest. >> >> I could understand if IO to disk for other processes became slow but >> everything including things like top just stop which is quite odd and >> concerning. > > Ok, new idea. Your disk activity is via the network, and your sessions > are via the network. Could it be that something network related is saturated? I saw this once before, a long time back, and every time I went through a debugging session, it came to some kind of lock on the sysctl tree with regards to the geom info (maybe the XML kind of tree dump or something). I don't recall all the details, but it was something like that. Eric From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:25:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31EA316A418; Fri, 1 Feb 2008 06:25:59 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 2414113C45D; Fri, 1 Feb 2008 06:25:57 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201846377; x=1202451177; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=ogsiAXWYKaWaPppUqNhh3 TJWoF+UW+WVd5mbzspDljE=; b=BPyZyD/oEobt0jMaPM3Ztk3N0gokjHBALCu9w YHIFlRqvPBJ48ftLB/tVG51DGLEVzAam+2cXYXV+esSBrp/VpzsLt9QJtYwuXZon oGqFSCUyUY5mwWBxHeJmOpXua+P8mS+O5t6Pg6+NKS1GD+/YH28iXiFo3DEyv3sF ckUOcM= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004970421.msg; Fri, 01 Feb 2008 06:12:54 +0000 Message-ID: <002201c86499$7861ac20$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" , "Dieter" References: <200801310147.BAA04522@sopwith.solgatos.com> <47A2A606.9080702@freebsd.org> Date: Fri, 1 Feb 2008 06:12:48 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 06:12:56 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 06:12:57 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:25:59 -0000 ----- Original Message ----- From: "Eric Anderson" > I saw this once before, a long time back, and every time I went through a debugging session, it came to some kind of lock on the > sysctl tree with regards to the geom info (maybe the XML kind of tree dump or something). I don't recall all the details, but > it was something like that. Yep thats where I've traced it to its requesting: kern.geom.confxml Which does:- static int sysctl_kern_geom_confxml(SYSCTL_HANDLER_ARGS) { int error; struct sbuf *sb; sb = sbuf_new(NULL, NULL, 0, SBUF_AUTOEXTEND); g_waitfor_event(g_confxml, sb, M_WAITOK, NULL); error = SYSCTL_OUT(req, sbuf_data(sb), sbuf_len(sb) + 1); sbuf_delete(sb); return error; } What I dont understand is why this would lock the entire machine. I've enabled LOCK_PROFILING and reran and I get the following which seems to indicate the culpret is: SYSCTL_LOCK() >From what I can tell g_waitfor_event is returning EAGAIN for a large amount of time which means we get stuck in:- userland_sysctl ... SYSCTL_LOCK(); do { req.oldidx = 0; req.newidx = 0; error = sysctl_root(0, name, namelen, &req); } while (error == EAGAIN); if (req.lock == REQ_WIRED && req.validlen > 0) vsunlock(req.oldptr, req.validlen); SYSCTL_UNLOCK(); ... The only reason I can see for returning EAGAIN is g_destroy_geom calling g_cancel_event To confirm this wasnt related to anything that top is doing and we are seeing a total lockup I used: sh -c 'while [ 1 ]; do sleep 1; date; done' Fri Feb 1 06:04:47 GMT 2008 Fri Feb 1 06:04:48 GMT 2008 Fri Feb 1 06:04:49 GMT 2008 <= Lockup Fri Feb 1 06:05:16 GMT 2008 <= Recover Fri Feb 1 06:05:17 GMT 2008 Fri Feb 1 06:05:18 GMT 2008 Enabling geom debugging I get:- Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, 0xffffff00010e6100, 2, 0) Feb 1 06:04:45 geomtest kernel: ref 0xffffff00010e6100 Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, 0xffffff00014e6700, 2, 0) Feb 1 06:04:45 geomtest kernel: ref 0xffffff00014e6700 Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, 0xffffff00010e6100, 2, 0) Feb 1 06:04:49 geomtest kernel: ref 0xffffff00010e6100 Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, 0xffffff00014e6700, 2, 0) Feb 1 06:04:49 geomtest kernel: ref 0xffffff00014e6700 Feb 1 06:04:49 geomtest kernel: mbr_taste(MBR,da0s3) Feb 1 06:04:49 geomtest kernel: g_mbrext_taste(MBREXT,da0s3) Feb 1 06:04:49 geomtest kernel: g_slice_spoiled(0xffffff0001b27180/da0s3) Feb 1 06:04:49 geomtest kernel: g_wither_geom(0xffffff0001a33c00(da0s3)) Feb 1 06:04:49 geomtest kernel: g_part_taste(PART,da0s3) Feb 1 06:04:56 geomtest kernel: g_post_event_x(0xffffffff80235b10, 0xffffff000144a9c0, 2, 262144) Feb 1 06:05:00 geomtest kernel: g_wither_geom(0xffffff000158b300(da0s3)) Feb 1 06:05:00 geomtest kernel: Feb 1 06:05:00 geomtest kernel: g_label_taste(LABEL, da0s3) Feb 1 06:05:00 geomtest kernel: Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: MSDOSFS: da0s3: FAT32 volume not valid. Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23980) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23980) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001b4be00(label:taste)) Feb 1 06:05:16 geomtest kernel: mbr_taste(MBR,da0s3g) Feb 1 06:05:16 geomtest kernel: g_slice_spoiled(0xffffff0001a3c280/da0s3g) Feb 1 06:05:16 geomtest kernel: g_wither_geom(0xffffff0001b4b600(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_mbrext_taste(MBREXT,da0s3g) Feb 1 06:05:16 geomtest kernel: bsd_taste(BSD,da0s3g) Feb 1 06:05:16 geomtest kernel: g_slice_spoiled(0xffffff0001b27200/da0s3g) Feb 1 06:05:16 geomtest kernel: g_wither_geom(0xffffff0001504c00(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_part_taste(PART,da0s3g) Feb 1 06:05:16 geomtest kernel: g_wither_geom(0xffffff0001527100(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_label_taste(LABEL, da0s3g) Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: UFS2 file system detected on da0s3g. Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: MSDOSFS: da0s3g: no FAT signature found. Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23a80) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23a80) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001b4be00(label:taste)) Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001a3c280) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001a3c280) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001b4b600(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b27180) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b27180) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001a33c00(da0s3)) Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b27200) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b27200) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001504c00(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001adc580) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001adc580) Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff0001527100(da0s3g)) Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23b00) Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23b00) Feb 1 06:05:16 geomtest kernel: Feb 1 06:05:16 geomtest kernel: g_destroy_geom(0xffffff000158b300(da0s3)) So after all that I can see why the sysctl call is taking so long to complete but the burning question is why does the entire system lock because of this? What else is waiting on the sysctl lock which is so critical? debug.lock.prof.stats: max total wait_total count avg wait_avg cnt_hold cnt_lock name 153585139 181478925 0 112 1620347 0 0 0 kern/vfs_subr.c:2062 (lockmgr:ufs) 26298436 26298513 23195820 24 1095771 966492 1 1 kern/kern_sysctl.c:1396 (sx:sysctl lock) 2396114 5604484 0 596 9403 0 0 0 sys/buf.h:280 (lockmgr:bufwait) 214 76577 1837 12490 6 0 185 280 kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain) 1 34126 2 30415 1 0 2 1 kern/kern_clock.c:224 (spin mutex:sched lock 1) 2 34069 5 30415 1 0 3 2 kern/kern_clock.c:224 (spin mutex:sched lock 0) 1 34067 0 30415 1 0 0 0 kern/kern_clock.c:264 (spin mutex:callout) 54 23629 1 4122 5 0 6 1 kern/uipc_socket.c:2468 (sleep mutex:so_snd) 34 22599 0 9232 2 0 0 0 amd64/amd64/pmap.c:935 (sleep mutex:vm page queue mutex) 34 22573 0 9232 2 0 0 0 amd64/amd64/pmap.c:936 (sleep mutex:pmap) 121 20165 0 586 34 0 0 0 kern/uipc_sockbuf.c:145 (sx:so_snd_sx) 50 19229 38 9401 2 0 3 3 geom/geom_io.c:68 (sleep mutex:bio queue) 117 16281 0 586 27 0 17 0 netinet/tcp_usrreq.c:781 (sleep mutex:inp) 50 14954 245 11968 1 0 15 62 kern/kern_conf.c:69 (sleep mutex:cdev) 52 13599 0 4122 3 0 0 0 kern/uipc_socket.c:2469 (sleep mutex:so_rcv) 227 11779 0 625 18 0 1 0 cam/cam_xpt.c:7153 (sleep mutex:arcmsr Q buffer lock) 40 11779 0 639 18 0 50 0 kern/kern_conf.c:372 (sleep mutex:Giant) 52 11201 219 3536 3 0 3 47 kern/kern_conf.c:384 (sleep mutex:Giant) 11 10888 0 1770 6 0 0 0 kern/subr_taskqueue.c:71 (spin mutex:fast_taskqueue) 41 10866 0 3041 3 0 0 0 netinet/tcp_subr.c:1426 (sleep mutex:isn_mtx) 50 9858 5 8243 1 0 1 3 kern/sys_generic.c:1079 (sleep mutex:sellck) 5 9545 7 3416 2 0 3 2 kern/sched_ule.c:1872 (spin mutex:sched lock 0) 2 9029 677 7356 1 0 103 319 kern/kern_timeout.c:419 (spin mutex:callout) 52 8916 0 1813 4 0 1 0 kern/sys_generic.c:759 (sleep mutex:sellck) 222 8318 0 625 13 0 0 0 geom/geom_disk.c:198 (sleep mutex:g_disk_done) 1 7843 3 6407 1 0 3 2 kern/kern_timeout.c:296 (spin mutex:callout) 5 7240 17 5090 1 0 5 7 kern/kern_switch.c:182 (spin mutex:sched lock 0) 64 6940 58 1276 5 0 0 3 kern/kern_timeout.c:241 (sleep mutex:Giant) 5 6924 2880 1730 4 1 2 1715 amd64/amd64/mp_machdep.c:966 (spin mutex:sched lock 0) 13 6845 0 623 10 0 0 0 amd64/amd64/mp_machdep.c:845 (spin mutex:smp rendezvous) 53 6279 0 1766 3 0 0 0 kern/sys_pipe.c:1331 (sleep mutex:pipe mutex) 5 6063 93 4189 1 0 2 27 kern/kern_intr.c:1131 (spin mutex:sched lock 0) 15 5910 25 660 8 0 7 10 kern/sys_generic.c:1127 (sleep mutex:sellck) 56 5290 727 357 14 2 0 17 netinet/tcp_input.c:479 (sleep mutex:inp) 144 5264 0 116 45 0 0 0 vm/vm_map.c:3111 (sx:user map) 7 4971 25 1552 3 0 9 11 kern/sched_ule.c:1872 (spin mutex:sched lock 1) 11 4959 0 625 7 0 0 0 dev/arcmsr/arcmsr.c:1129 (sleep mutex:arcmsr Q buffer lock) 210 4901 56 3436 1 0 9 27 kern/sched_ule.c:979 (spin mutex:sched lock 0) 69 4899 0 357 13 0 0 0 netinet/tcp_input.c:400 (sleep mutex:tcp) 1 4698 0 4045 1 0 0 0 kern/kern_clock.c:412 (spin mutex:sched lock 1) 1 4697 0 4189 1 0 1 0 kern/kern_intr.c:706 (spin mutex:sched lock 0) 1 4696 0 4184 1 0 0 0 kern/kern_timeout.c:191 (spin mutex:callout) 45 4604 0 591 7 0 6 0 kern/vfs_bio.c:3790 (sleep mutex:bdone lock) 2 4550 3 4045 1 0 0 1 kern/kern_clock.c:412 (spin mutex:sched lock 0) 53 4384 0 1228 3 0 0 0 kern/sys_generic.c:750 (sleep mutex:sellck) 2 4189 3 3392 1 0 1 2 kern/subr_sleepqueue.c:331 (spin mutex:sched lock 0) 3695 4042 0 4 1010 0 0 0 kern/vfs_vnops.c:515 (lockmgr:ufs) 8 3891 0 645 6 0 0 0 kern/subr_sleepqueue.c:409 (spin mutex:sleepq chain) 10 3773 0 3103 1 0 1 0 kern/vfs_bio.c:3002 (sleep mutex:bdone lock) 55 3323 22 634 5 0 4 7 kern/kern_mutex.c:141 (sleep mutex:sellck) 50 3188 0 2469 1 0 0 0 kern/kern_descrip.c:2139 (sleep mutex:sleep mtxpool) 1853 3083 0 4 770 0 0 0 kern/vfs_subr.c:1665 (lockmgr:syncer) 6 3074 0 1310 2 0 0 0 kern/kern_lock.c:200 (sleep mutex:lockbuilder mtxpool) 10 3003 0 645 4 0 0 0 kern/subr_sleepqueue.c:390 (sleep mutex:process lock) 32 2991 0 2448 1 0 0 0 kern/kern_descrip.c:2007 (sleep mutex:sleep mtxpool) 50 2852 0 2401 1 0 0 0 kern/kern_sig.c:996 (sleep mutex:process lock) 94 2458 0 589 4 0 0 0 dev/em/if_em.c:980 (sleep mutex:em0) 54 2358 6 592 3 0 0 3 kern/kern_conf.c:360 (sleep mutex:Giant) 1 2343 35 1547 1 0 20 20 kern/subr_sleepqueue.c:331 (spin mutex:sched lock 1) 2 2272 98 1540 1 0 238 39 kern/sched_ule.c:979 (spin mutex:sched lock 1) 1 2214 0 1897 1 0 0 0 kern/subr_sleepqueue.c:595 (spin mutex:sched lock 0) 1 2194 4 1329 1 0 2 2 kern/kern_timeout.c:500 (spin mutex:callout) 1 2189 33 1526 1 0 0 4 kern/subr_sleepqueue.c:800 (spin mutex:sleepq chain) 5 2160 1959 1173 1 1 0 1168 amd64/amd64/mp_machdep.c:966 (spin mutex:sched lock 1) 1 2158 1 1660 1 0 0 1 kern/sys_generic.c:753 (spin mutex:sched lock 0) 8 1972 0 248 7 0 0 0 kern/kern_timeout.c:241 (sleep mutex:process lock) 6 1968 11 1245 1 0 2 5 kern/kern_intr.c:1131 (spin mutex:sched lock 1) ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:38:09 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 685F716A41B for ; Fri, 1 Feb 2008 06:38:09 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0130B13C4CE for ; Fri, 1 Feb 2008 06:38:08 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m115qT7T067150; Thu, 31 Jan 2008 21:52:29 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m115qSf7075944; Thu, 31 Jan 2008 21:52:28 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (61.204.211.246.customerlink.pwd.ne.jp [61.204.211.246]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m115qSCd024190; Thu, 31 Jan 2008 21:52:28 -0800 (PST) (envelope-from gnn@neville-neil.com) Date: Fri, 01 Feb 2008 14:52:27 +0900 Message-ID: From: gnn@freebsd.org To: Stefan Lambrev In-Reply-To: <47A0B023.5020401@moneybookers.com> References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.10.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:38:09 -0000 At Wed, 30 Jan 2008 19:13:07 +0200, Stefan Lambrev wrote: > > Greetings, > > After playing with many settings and testing various configuration, now > I'm able to to receive on bridge more then 800,000 packets/s > without errors, which is amazing! > Unfortunately the server behind bridge can't handle more then 250,000 > packets/s > Please advise how I can increase those limits? > Is is possible? > > The servers are with 82573E Gigabit Ethernet Controller (quad port) > So far I tried with lagg and ng_fec, but with them I see more problems > then benefits :) > Tried polling with kern.polling.user_frac from 5 to 95, > different HZ, but nothing helped. Increase the size of your socket buffers. Increase the amount of mbufs in the system. Best, George From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:38:17 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FF2116A41A for ; Fri, 1 Feb 2008 06:38:17 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id 460CE13C4E5 for ; Fri, 1 Feb 2008 06:38:17 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m115t37T067225; Thu, 31 Jan 2008 21:55:03 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m115t2Iu076669; Thu, 31 Jan 2008 21:55:02 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (61.204.211.246.customerlink.pwd.ne.jp [61.204.211.246]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m115t1Yv024529; Thu, 31 Jan 2008 21:55:02 -0800 (PST) (envelope-from gnn@neville-neil.com) Date: Fri, 01 Feb 2008 14:55:01 +0900 Message-ID: From: gnn@freebsd.org To: Erik Cederstrand In-Reply-To: <479745DA.8010003@cederstrand.dk> References: <4796C717.9000507@cederstrand.dk> <47972895.4050005@FreeBSD.org> <479745DA.8010003@cederstrand.dk> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.10.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: Kris Kennaway , "freebsd-performance@freebsd.org" Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:38:17 -0000 At Wed, 23 Jan 2008 14:49:14 +0100, Erik Cederstrand wrote: > > Kris Kennaway wrote: > > > > This is coming along very nicely indeed! > > > > One suggestion I have is that as more metrics are added it becomes > > important for an "at a glance" overview of changes so we can monitor for > > performance improvements and regressions among many workloads. > > > > One way to do this would be a matrix of each metric with its change > > compared to recent samples. e.g. you could do a student's T comparison > > of today's numbers with those from yesterday, or from a week ago, and > > colour-code those that show a significant deviation from "no change". > > This might be a bit noisy on short timescales, so you could aggregrate > > data into larger bins and compare e.g. moving 1-week aggregates. > > Fluctuations on short timescales won't stand out, but if there is a real > > change then it will show up less than a week later. > > I agree that there's a need for an overview and some sort of > notification. I've been collecting historical data to get a baseline for > the statistics and I'll try to see what I can do over the next weeks. > > > These significant events could also be graphed themselves and/or a > > history log maintained (or automatically annotated on the individual > > graphs) so historical changes can also be pinpointed. > > > > At some point the ability to annotate the data will become important > > (e.g. "We understand the cause of this, it was r1.123 of foo.c, which > > was corrected in r1.124. The developer responsible has been shot.") > > There's a field in the database for this sort of thing. I just think it > needs some sort of authentication. That'll have to wait a bit. > > > P.S. If I understand correctly, the float test shows a regression? The > > metric is calculations/second, so higher = better? > > The documentation on Unixbench is scarce, but I would think so. > > BTW if anyone's interested my SVN repo is online at: > > svn://littlebit.dk/website/trunk (Pylons project) > svn://littlebit.dk/tracker/trunk (sh/Python scripts for runnning the > server and slaves) > > Be careful with your eyes - this is my first attempt at both shell > scripting and Python :-) > BTW This is excellent work and, if I get the time, I'll be using some of this at my day job. Thanks, George From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 07:07:51 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38C7816A41A for ; Fri, 1 Feb 2008 07:07:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outO.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id 1C9D413C45D for ; Fri, 1 Feb 2008 07:07:50 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Thu, 31 Jan 2008 23:07:50 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id B23E812708F; Thu, 31 Jan 2008 23:07:49 -0800 (PST) Message-ID: <47A2C544.4090303@elischer.org> Date: Thu, 31 Jan 2008 23:07:48 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> In-Reply-To: <47A2C2A2.5040109@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 07:07:51 -0000 Alexander Motin wrote: > Julian Elischer пишет: >> Alexander Motin wrote: >>> Hi. >>> >>> While profiling netgraph operation on UP HEAD router I have found >>> that huge amount of time it spent on memory allocation/deallocation: >>> >>> 0.14 0.05 132119/545292 ip_forward [12] >>> 0.14 0.05 133127/545292 fxp_add_rfabuf [18] >>> 0.27 0.10 266236/545292 ng_package_data [17] >>> [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] >>> 0.17 0.00 545292/1733401 critical_exit [98] >>> 0.01 0.00 275941/679675 generic_bzero [68] >>> 0.01 0.00 133127/133127 mb_ctor_pack [103] >>> >>> 0.15 0.06 133100/545266 mb_free_ext [22] >>> 0.15 0.06 133121/545266 m_freem [15] >>> 0.29 0.11 266236/545266 ng_free_item [16] >>> [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] >>> 0.17 0.00 545266/1733401 critical_exit [98] >>> 0.00 0.04 133100/133100 mb_dtor_pack [57] >>> 0.00 0.00 134121/134121 mb_dtor_mbuf [111] >>> >>> I have already optimized all possible allocation calls and those that >>> left are practically unavoidable. But even after this kgmon tells >>> that 30% of CPU time consumed by memory management. >>> >>> So I have some questions: >>> 1) Is it real situation or just profiler mistake? >>> 2) If it is real then why UMA is so slow? I have tried to replace it >>> in some places with preallocated TAILQ of required memory blocks >>> protected by mutex and according to profiler I have got _much_ better >>> results. Will it be a good practice to replace relatively small UMA >>> zones with preallocated queue to avoid part of UMA calls? >>> 3) I have seen that UMA does some kind of CPU cache affinity, but >>> does it cost so much that it costs 30% CPU time on UP router? >> >> given this information, I would add an 'item cache' in ng_base.c >> (hmm do I already have one?) > > That was actually my second question. As there is only 512 items by > default and they are small in size I can easily preallocate them all on > boot. But is it a good way? Why UMA can't do just the same when I have > created zone with specified element size and maximum number of objects? > What is the principal difference? > who knows what uma does.. but if you do it yourself you know what the overhead is.. :-) From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 08:24:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C50516A41A for ; Fri, 1 Feb 2008 08:24:16 +0000 (UTC) (envelope-from kometen@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id D8D0813C46A for ; Fri, 1 Feb 2008 08:24:15 +0000 (UTC) (envelope-from kometen@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so296377anc.13 for ; Fri, 01 Feb 2008 00:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=dTdsI3lfeoCKIKNNM3ghUpEK5biONX2Ipu2WRGqKf4s=; b=oZdQWFghT4OO2iVudFzC3Ftmy0RUQJebQuNeveikIEKHt796SXUNrl2Q+8GegnPHvIGfCWAQcYQkjbig/OoO6RmQOwuBIiSaBlwgPCQQhZHx2QhsSFdYx9Ivv6PdxETHU7LLL1MzL9ijGbF+/2kHTAAaOz1XUsd7yw1ZqxnChVk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=JBSYHHqulwgqsOJHQuVCNEEV9AC9nLIld//wMnqB8EcPUMBY+t6Tf0huXq1MXjUv4Tm+nNhfr2DhSgkRp8sPztXW66nark2lOjKmqsMF6zueJicLD55cjvZYWHer6hKeeo0a6qSWkARAifsPNk700iL4IyaN8/QAnfJb5ZcLHX0= Received: by 10.100.249.9 with SMTP id w9mr6713722anh.67.1201854255208; Fri, 01 Feb 2008 00:24:15 -0800 (PST) Received: by 10.100.242.10 with HTTP; Fri, 1 Feb 2008 00:24:15 -0800 (PST) Message-ID: Date: Fri, 1 Feb 2008 09:24:15 +0100 From: "Claus Guttesen" To: "Niki Denev" In-Reply-To: <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <479E3C5E.1070405@FreeBSD.org> <47A043FD.9090607@FreeBSD.org> <47A0CD11.802@FreeBSD.org> <9bbcef730801301157n25684018y3d43bfdd0301a8dd@mail.gmail.com> <2e77fc10801310826v2fabb5g594840241784fc7@mail.gmail.com> Cc: freebsd-performance@freebsd.org Subject: Re: postgresql-performance using sysbench X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 08:24:16 -0000 > Just a thought on the effect that HZ has on filesystem (and overall) > performance : > Linux has sort of backtracked from defaulting to HZ=1000 and enable it > only on kernels compiled > for "Desktop" work, and setting HZ=250 for the "Server" profile. I'm doing some db-imports on postgresql on a file which takes 3 hours and 40 min. with default HZ (1000). Then I changed HZ to 250 (in loader.conf) and rebooted. The import still took 3 h. 40 min. so altering HZ does not seem to have any influence. #>sysctl kern.clockrate kern.clockrate: { hz = 250, tick = 4000, profhz = 166, stathz = 33 } -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:51:20 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49D5E16A41B; Fri, 1 Feb 2008 06:51:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4CC3113C45B; Fri, 1 Feb 2008 06:51:18 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71506898; Fri, 01 Feb 2008 08:51:18 +0200 Message-ID: <47A2C142.9030306@FreeBSD.org> Date: Fri, 01 Feb 2008 08:50:42 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <47A25412.3010301@FreeBSD.org> <47A255BB.3020708@FreeBSD.org> <47A25A3A.80700@FreeBSD.org> In-Reply-To: <47A25A3A.80700@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 01 Feb 2008 08:32:08 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:51:20 -0000 Kris Kennaway пишет: > Alexander Motin wrote: >> Alexander Motin пишет: >>> While profiling netgraph operation on UP HEAD router I have found >>> that huge amount of time it spent on memory allocation/deallocation: >> >> I have forgotten to tell that it was mostly GENERIC kernel just built >> without INVARIANTS, WITNESS and SMP but with 'profile 2'. > > What is 'profile 2'? I have thought it is high resolution profiling support. Isn't it? -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:57:11 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 523EA16A419; Fri, 1 Feb 2008 06:57:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 96A9E13C43E; Fri, 1 Feb 2008 06:57:10 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71507531; Fri, 01 Feb 2008 08:57:09 +0200 Message-ID: <47A2C2A2.5040109@FreeBSD.org> Date: Fri, 01 Feb 2008 08:56:34 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Julian Elischer References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> In-Reply-To: <47A25A0D.2080508@elischer.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 01 Feb 2008 08:32:08 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:57:11 -0000 Julian Elischer пишет: > Alexander Motin wrote: >> Hi. >> >> While profiling netgraph operation on UP HEAD router I have found that >> huge amount of time it spent on memory allocation/deallocation: >> >> 0.14 0.05 132119/545292 ip_forward [12] >> 0.14 0.05 133127/545292 fxp_add_rfabuf [18] >> 0.27 0.10 266236/545292 ng_package_data [17] >> [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] >> 0.17 0.00 545292/1733401 critical_exit [98] >> 0.01 0.00 275941/679675 generic_bzero [68] >> 0.01 0.00 133127/133127 mb_ctor_pack [103] >> >> 0.15 0.06 133100/545266 mb_free_ext [22] >> 0.15 0.06 133121/545266 m_freem [15] >> 0.29 0.11 266236/545266 ng_free_item [16] >> [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] >> 0.17 0.00 545266/1733401 critical_exit [98] >> 0.00 0.04 133100/133100 mb_dtor_pack [57] >> 0.00 0.00 134121/134121 mb_dtor_mbuf [111] >> >> I have already optimized all possible allocation calls and those that >> left are practically unavoidable. But even after this kgmon tells that >> 30% of CPU time consumed by memory management. >> >> So I have some questions: >> 1) Is it real situation or just profiler mistake? >> 2) If it is real then why UMA is so slow? I have tried to replace it >> in some places with preallocated TAILQ of required memory blocks >> protected by mutex and according to profiler I have got _much_ better >> results. Will it be a good practice to replace relatively small UMA >> zones with preallocated queue to avoid part of UMA calls? >> 3) I have seen that UMA does some kind of CPU cache affinity, but does >> it cost so much that it costs 30% CPU time on UP router? > > given this information, I would add an 'item cache' in ng_base.c > (hmm do I already have one?) That was actually my second question. As there is only 512 items by default and they are small in size I can easily preallocate them all on boot. But is it a good way? Why UMA can't do just the same when I have created zone with specified element size and maximum number of objects? What is the principal difference? -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 10:26:34 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EBBC16A41A; Fri, 1 Feb 2008 10:26:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 77E2D13C468; Fri, 1 Feb 2008 10:26:33 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A2F3D7.1020307@FreeBSD.org> Date: Fri, 01 Feb 2008 11:26:31 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> <47A255BB.3020708@FreeBSD.org> <47A25A3A.80700@FreeBSD.org> <47A2C142.9030306@FreeBSD.org> In-Reply-To: <47A2C142.9030306@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 10:26:34 -0000 Alexander Motin wrote: > Kris Kennaway пишет: >> Alexander Motin wrote: >>> Alexander Motin пишет: >>>> While profiling netgraph operation on UP HEAD router I have found >>>> that huge amount of time it spent on memory allocation/deallocation: >>> >>> I have forgotten to tell that it was mostly GENERIC kernel just built >>> without INVARIANTS, WITNESS and SMP but with 'profile 2'. >> >> What is 'profile 2'? > > I have thought it is high resolution profiling support. Isn't it? > OK. This is not commonly used so I don't know if it works. Try using hwpmc if possible to compare. When you say that your own allocation routines show less time use under profiling, how do they affect the actual system performance? Kris From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 11:29:25 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E24D316A46C for ; Fri, 1 Feb 2008 11:29:25 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 6AC8C13C478 for ; Fri, 1 Feb 2008 11:29:25 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JKu5E-0001Qf-6W for freebsd-performance@freebsd.org; Fri, 01 Feb 2008 11:29:24 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 01 Feb 2008 11:29:24 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 01 Feb 2008 11:29:24 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Fri, 01 Feb 2008 12:30:02 +0100 Lines: 128 Message-ID: References: <200801310147.BAA04522@sopwith.solgatos.com> <47A2A606.9080702@freebsd.org> <002201c86499$7861ac20$b6db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig40EA324C8EC7A4591F4DDCAD" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <002201c86499$7861ac20$b6db87d4@multiplay.co.uk> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 11:29:26 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig40EA324C8EC7A4591F4DDCAD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Steven Hartland wrote: > Yep thats where I've traced it to its requesting: kern.geom.confxml >=20 > Which does:- > static int > sysctl_kern_geom_confxml(SYSCTL_HANDLER_ARGS) > { > int error; > struct sbuf *sb; >=20 > sb =3D sbuf_new(NULL, NULL, 0, SBUF_AUTOEXTEND); > g_waitfor_event(g_confxml, sb, M_WAITOK, NULL); > error =3D SYSCTL_OUT(req, sbuf_data(sb), sbuf_len(sb) + 1); > sbuf_delete(sb); > return error; > } >=20 > What I dont understand is why this would lock the entire machine. >=20 > I've enabled LOCK_PROFILING and reran and I get the following which > seems to indicate the culpret is: SYSCTL_LOCK() >=20 >> From what I can tell g_waitfor_event is returning EAGAIN for a large > amount of time which means we get stuck in:- > userland_sysctl > ... > SYSCTL_LOCK(); >=20 > do { > req.oldidx =3D 0; > req.newidx =3D 0; > error =3D sysctl_root(0, name, namelen, &req); > } while (error =3D=3D EAGAIN); >=20 > if (req.lock =3D=3D REQ_WIRED && req.validlen > 0) > vsunlock(req.oldptr, req.validlen); >=20 > SYSCTL_UNLOCK(); > ... >=20 > The only reason I can see for returning EAGAIN is g_destroy_geom > calling g_cancel_event >=20 > To confirm this wasnt related to anything that top is doing and > we are seeing a total lockup I used: > sh -c 'while [ 1 ]; do sleep 1; date; done' > Fri Feb 1 06:04:47 GMT 2008 > Fri Feb 1 06:04:48 GMT 2008 > Fri Feb 1 06:04:49 GMT 2008 <=3D Lockup > Fri Feb 1 06:05:16 GMT 2008 <=3D Recover > Fri Feb 1 06:05:17 GMT 2008 > Fri Feb 1 06:05:18 GMT 2008 >=20 >=20 > Enabling geom debugging I get:- > Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, > 0xffffff00010e6100, 2, 0) > Feb 1 06:04:45 geomtest kernel: ref 0xffffff00010e6100 > Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, > 0xffffff00014e6700, 2, 0) > Feb 1 06:04:45 geomtest kernel: ref 0xffffff00014e6700 > Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, > 0xffffff00010e6100, 2, 0) > Feb 1 06:04:49 geomtest kernel: ref 0xffffff00010e6100 > Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, > 0xffffff00014e6700, 2, 0) > Feb 1 06:04:49 geomtest kernel: ref 0xffffff00014e6700 > Feb 1 06:04:49 geomtest kernel: mbr_taste(MBR,da0s3) > Feb 1 06:04:49 geomtest kernel: g_mbrext_taste(MBREXT,da0s3) > Feb 1 06:04:49 geomtest kernel: g_slice_spoiled(0xffffff0001b27180/da0= s3) > Feb 1 06:04:49 geomtest kernel: g_wither_geom(0xffffff0001a33c00(da0s3= )) > Feb 1 06:04:49 geomtest kernel: g_part_taste(PART,da0s3) > Feb 1 06:04:56 geomtest kernel: g_post_event_x(0xffffffff80235b10, > 0xffffff000144a9c0, 2, 262144) > Feb 1 06:05:00 geomtest kernel: g_wither_geom(0xffffff000158b300(da0s3= )) > Feb 1 06:05:00 geomtest kernel: > Feb 1 06:05:00 geomtest kernel: g_label_taste(LABEL, da0s3) > Feb 1 06:05:00 geomtest kernel: > Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: MSDOSFS: da0s3: FAT32 > volume not valid. > Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23980) > Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23980)= > Feb 1 06:05:16 geomtest kernel: > So after all that I can see why the sysctl call is taking > so long to complete but the burning question is why does Can you explain - I don't see it :) Do you mean to say there's a contention for sysctl lock between geom_confxml and g_waitfor_event or that geom_label tasting has something to do with it? > the entire system lock because of this? What else is > waiting on the sysctl lock which is so critical? What I do know is that sysctl is GIANT-locked, which is also used by some parts of device handling infrastructure (dead_cdevsw), the USB stack, and can creep itself in the timer via swi_sched in subr_taskqueue.c:303. I cannot say for sure that's what happening here, but they are possibilities. If you can provoke this reliably, I think there is a (old!) patch for removing sysctl from under the Giant lock that you could try. --------------enig40EA324C8EC7A4591F4DDCAD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHowLAldnAQVacBcgRAkutAJ0Y3W1r/pUDkumTH7lwOQGEPJAVwgCfUQ+j GrktnsKT+fVbSWCNowj0Y0E= =uomb -----END PGP SIGNATURE----- --------------enig40EA324C8EC7A4591F4DDCAD-- From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 11:49:39 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64A3E16A420 for ; Fri, 1 Feb 2008 11:49:39 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id A492113C45D for ; Fri, 1 Feb 2008 11:49:38 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id E03531B10EF4; Fri, 1 Feb 2008 12:49:36 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, J_CHICKENPOX_48 autolearn=no version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id BBF151B10EF0; Fri, 1 Feb 2008 12:49:33 +0100 (CET) Message-ID: <47A3074A.3040409@moneybookers.com> Date: Fri, 01 Feb 2008 13:49:30 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: gnn@freebsd.org References: <4794E6CC.1050107@moneybookers.com> <47A0B023.5020401@moneybookers.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/5631/Fri Feb 1 11:41:51 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: network performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 11:49:39 -0000 Greetings, gnn@freebsd.org wrote: > At Wed, 30 Jan 2008 19:13:07 +0200, > Stefan Lambrev wrote: > >> Greetings, >> >> After playing with many settings and testing various configuration, now >> I'm able to to receive on bridge more then 800,000 packets/s >> without errors, which is amazing! >> Unfortunately the server behind bridge can't handle more then 250,000 >> packets/s >> Please advise how I can increase those limits? >> Is is possible? >> >> The servers are with 82573E Gigabit Ethernet Controller (quad port) >> So far I tried with lagg and ng_fec, but with them I see more problems >> then benefits :) >> Tried polling with kern.polling.user_frac from 5 to 95, >> different HZ, but nothing helped. >> > > Increase the size of your socket buffers. > > Increase the amount of mbufs in the system. > > Best, > George > Here is what I put in my sysctl.conf: kern.random.sys.harvest.ethernet=0 kern.ipc.nmbclusters=262144 kern.ipc.maxsockbuf=2097152 kern.ipc.maxsockets=98624 kern.ipc.somaxconn=1024 and in /boot/loader.conf: vm.kmem_size="1024M" kern.hz="500" this is from netstat -m 516/774/1290 mbufs in use (current/cache/total) 513/411/924/262144 mbuf clusters in use (current/cache/total/max) 513/383 mbuf+clusters out of packet secondary zone in use (current/cache) 0/2/2/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 1155K/1023K/2178K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines But still netstat -w1 -I em0 shows: input (em0) output packets errs bytes packets errs bytes colls 273877 113313 16432620 254270 0 14746500 0 273397 109905 16403820 253946 0 14728810 0 273945 113337 16436700 254285 0 14750560 0 What bothers me is the output of top -S: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 22 root 1 -68 - 0K 16K CPU1 1 12:11 100.00% em0 taskq 11 root 1 171 ki31 0K 16K RUN 0 21:56 99.17% idle: cpu0 10 root 1 171 ki31 0K 16K RUN 1 9:16 0.00% idle: cpu1 14 root 1 -44 - 0K 16K WAIT 0 0:07 0.00% swi1: net and vmstat: procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad4 in sy cs us sy id 1 0 0 67088 1939700 0 0 0 0 0 0 0 2759 119 1325 0 50 50 0 0 0 67088 1939700 0 0 0 0 0 0 0 2760 127 1178 0 50 50 0 0 0 67088 1939700 0 0 0 0 0 0 0 2761 120 1269 0 50 50 What I'm missing? From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 12:21:00 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA6A816A417 for ; Fri, 1 Feb 2008 12:21:00 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 9F24413C4CE for ; Fri, 1 Feb 2008 12:21:00 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.tenantsolutions.net [209.163.168.124] (may be forged)) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id m11CKtqD071587; Fri, 1 Feb 2008 06:20:58 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <47A30EA7.7050506@freebsd.org> Date: Fri, 01 Feb 2008 06:20:55 -0600 From: Eric Anderson User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Steven Hartland References: <200801310147.BAA04522@sopwith.solgatos.com> <47A2A606.9080702@freebsd.org> <002201c86499$7861ac20$b6db87d4@multiplay.co.uk> In-Reply-To: <002201c86499$7861ac20$b6db87d4@multiplay.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: Dieter , freebsd-performance@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 12:21:00 -0000 Steven Hartland wrote: > > ----- Original Message ----- From: "Eric Anderson" > >> I saw this once before, a long time back, and every time I went >> through a debugging session, it came to some kind of lock on the >> sysctl tree with regards to the geom info (maybe the XML kind of tree >> dump or something). I don't recall all the details, but it was >> something like that. > > Yep thats where I've traced it to its requesting: kern.geom.confxml > > Which does:- > static int > sysctl_kern_geom_confxml(SYSCTL_HANDLER_ARGS) > { > int error; > struct sbuf *sb; > > sb = sbuf_new(NULL, NULL, 0, SBUF_AUTOEXTEND); > g_waitfor_event(g_confxml, sb, M_WAITOK, NULL); > error = SYSCTL_OUT(req, sbuf_data(sb), sbuf_len(sb) + 1); > sbuf_delete(sb); > return error; > } > > What I dont understand is why this would lock the entire machine. > > I've enabled LOCK_PROFILING and reran and I get the following which > seems to indicate the culpret is: SYSCTL_LOCK() > > From what I can tell g_waitfor_event is returning EAGAIN for a large > amount of time which means we get stuck in:- > userland_sysctl > ... > SYSCTL_LOCK(); > > do { > req.oldidx = 0; > req.newidx = 0; > error = sysctl_root(0, name, namelen, &req); > } while (error == EAGAIN); > > if (req.lock == REQ_WIRED && req.validlen > 0) > vsunlock(req.oldptr, req.validlen); > > SYSCTL_UNLOCK(); > ... > > The only reason I can see for returning EAGAIN is g_destroy_geom > calling g_cancel_event Wait - if it returns EAGAIN for a while, then look at that code above. It will hold the sysctl lock for some indefinite amount of time. Maybe it should look like this instead: do { SYSCTL_LOCK(); req.oldidx = 0; req.newidx = 0; error = sysctl_root(0, name, namelen, &req); SYSCTL_UNLOCK(); } while (error == EAGAIN); if (req.lock == REQ_WIRED && req.validlen > 0) vsunlock(req.oldptr, req.validlen); Can you try that? Eric From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 12:26:40 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 713DA16A41A for ; Fri, 1 Feb 2008 12:26:40 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id CFF0C13C467 for ; Fri, 1 Feb 2008 12:26:39 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201868606; x=1202473406; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=gYmfS3sPuwK7qiFAS1wEi 83iwqAJQZF6GSfVvDwFWlw=; b=HYDx2FtsPN4OHhDG6POGIXPr5QwYONgZkEt+n jDb1dxBOttyPUHq5JxQ0D7yvKg6UAgZf9swUupxxWhA914r1f2ifVRAgVmxb9Q4a CBdgplWMciLH0w1VxeCfI8sugCwWMCfwzsjEFzotn9PycWeKGZ5F332btinXn8vk 7H9fvk= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004973364.msg; Fri, 01 Feb 2008 12:23:24 +0000 Message-ID: <012401c864cd$3c0c9530$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: , "Ivan Voras" References: <200801310147.BAA04522@sopwith.solgatos.com> <47A2A606.9080702@freebsd.org><002201c86499$7861ac20$b6db87d4@multiplay.co.uk> Date: Fri, 1 Feb 2008 12:23:21 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 12:23:25 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 12:23:26 +0000 Cc: Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 12:26:40 -0000 ----- Original Message ----- From: "Ivan Voras" ... >> geom debugging I get:- >> Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, >> 0xffffff00010e6100, 2, 0) >> Feb 1 06:04:45 geomtest kernel: ref 0xffffff00010e6100 >> Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, >> 0xffffff00014e6700, 2, 0) >> Feb 1 06:04:45 geomtest kernel: ref 0xffffff00014e6700 >> Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, >> 0xffffff00010e6100, 2, 0) >> Feb 1 06:04:49 geomtest kernel: ref 0xffffff00010e6100 >> Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, >> 0xffffff00014e6700, 2, 0) >> Feb 1 06:04:49 geomtest kernel: ref 0xffffff00014e6700 >> Feb 1 06:04:49 geomtest kernel: mbr_taste(MBR,da0s3) >> Feb 1 06:04:49 geomtest kernel: g_mbrext_taste(MBREXT,da0s3) >> Feb 1 06:04:49 geomtest kernel: g_slice_spoiled(0xffffff0001b27180/da0s3) >> Feb 1 06:04:49 geomtest kernel: g_wither_geom(0xffffff0001a33c00(da0s3)) >> Feb 1 06:04:49 geomtest kernel: g_part_taste(PART,da0s3) >> Feb 1 06:04:56 geomtest kernel: g_post_event_x(0xffffffff80235b10, >> 0xffffff000144a9c0, 2, 262144) >> Feb 1 06:05:00 geomtest kernel: g_wither_geom(0xffffff000158b300(da0s3)) >> Feb 1 06:05:00 geomtest kernel: >> Feb 1 06:05:00 geomtest kernel: g_label_taste(LABEL, da0s3) >> Feb 1 06:05:00 geomtest kernel: >> Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: MSDOSFS: da0s3: FAT32 >> volume not valid. >> Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23980) >> Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23980) >> Feb 1 06:05:16 geomtest kernel: > >> So after all that I can see why the sysctl call is taking >> so long to complete but the burning question is why does > > Can you explain - I don't see it :) Do you mean to say there's a > contention for sysctl lock between geom_confxml and g_waitfor_event or > that geom_label tasting has something to do with it? Nope what I belive to be happening is this sysctl_kern_geom_confxml which is called while SYSCTL_LOCK is held from userland_sysctl returns EAGAIN for a considerable period while newfs runs. >> the entire system lock because of this? What else is >> waiting on the sysctl lock which is so critical? > > What I do know is that sysctl is GIANT-locked, which is also used by > some parts of device handling infrastructure (dead_cdevsw), the USB > stack, and can creep itself in the timer via swi_sched in > subr_taskqueue.c:303. I cannot say for sure that's what happening here, > but they are possibilities. > > If you can provoke this reliably, I think there is a (old!) patch for > removing sysctl from under the Giant lock that you could try. Yes this is 100% reproducable and I'm beginning to suspect this issue may the cause of the regular pauses on our hosting machines so it might have further reaching effects than just this one example. If SYSCTL_LOCK is indeed GIANT locked then it might explain why its effect is so pronounced. Tried ADAPTIVE_SX as test to see if that helped as the sysctl lock is and sx but didnt help Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 13:32:45 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91BD016A41A; Fri, 1 Feb 2008 13:32:45 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id EB16813C442; Fri, 1 Feb 2008 13:32:44 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201872563; x=1202477363; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=6XqYJb+iHQ6yw6HJ8HCRJ Uau0+faLV2yFNvdgflp5ik=; b=XQxhi3ts9y8xWM5G24JDFeL4W7pUMLNTzBnrE bBN+O91TxDmcJUJbIl1H++yX7Rejkcqn5zNp+5zpa5LL4MKH7VYwvbT0W+WUmF8u kr3eOBUQFK/8Z88Z3hqgE2Q+EsU9+iozbAOGAkfS/uaKOiCooe+TlUvataL+6Wd4 xGdWNc= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004973734.msg; Fri, 01 Feb 2008 13:29:22 +0000 Message-ID: <016401c864d6$73251c50$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" References: <200801310147.BAA04522@sopwith.solgatos.com><47A2A606.9080702@freebsd.org><002201c86499$7861ac20$b6db87d4@multiplay.co.uk> <47A30EA7.7050506@freebsd.org> Date: Fri, 1 Feb 2008 13:29:19 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 13:29:23 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 13:29:23 +0000 Cc: Dieter , freebsd-performance@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 13:32:45 -0000 ----- Original Message ----- From: "Eric Anderson" > Wait - if it returns EAGAIN for a while, then look at that code above. > It will hold the sysctl lock for some indefinite amount of time. Maybe > it should look like this instead: > > > do { > SYSCTL_LOCK(); > req.oldidx = 0; > req.newidx = 0; > error = sysctl_root(0, name, namelen, &req); > SYSCTL_UNLOCK(); > } while (error == EAGAIN); > > if (req.lock == REQ_WIRED && req.validlen > 0) > vsunlock(req.oldptr, req.validlen); > Tried no difference unfortunately, possibly its give the closeness of the lock unlock its just reacquiring the lock straight after releasing it hence not giving the other thread chance to run / obtain the lock? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 13:58:07 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D539416A418; Fri, 1 Feb 2008 13:58:07 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 3E0D113C448; Fri, 1 Feb 2008 13:58:07 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201874238; x=1202479038; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=ELQeJzXmQpJ6PKdXj5yTI KBtVxGm3CKtYQVKJwRgitk=; b=lquxXSELur70yJnO/eh0LxcBSpXdXKJZRTlN3 1uV/zWImFUYUJivGrgedHoB0z5RXoiMdM0BWXgjbHPLRrT7LnPPnWd3PYtFmWjeV s1YP9IucF6e+//F09nWG5vOU1feqHhC9Ge8ESsyQl8mU1QrRo5Qm3G/+vcLkSzk3 jYHDYI= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004973894.msg; Fri, 01 Feb 2008 13:57:15 +0000 Message-ID: <018701c864da$57020f20$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Eric Anderson" References: <200801310147.BAA04522@sopwith.solgatos.com><47A2A606.9080702@freebsd.org><002201c86499$7861ac20$b6db87d4@multiplay.co.uk><47A30EA7.7050506@freebsd.org> <016401c864d6$73251c50$b6db87d4@multiplay.co.uk> Date: Fri, 1 Feb 2008 13:57:09 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 13:57:16 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 13:57:18 +0000 Cc: Dieter , freebsd-performance@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 13:58:07 -0000 ----- Original Message ----- From: "Steven Hartland" > ----- Original Message ----- > From: "Eric Anderson" >> Wait - if it returns EAGAIN for a while, then look at that code above. >> It will hold the sysctl lock for some indefinite amount of time. Maybe >> it should look like this instead: >> >> >> do { >> SYSCTL_LOCK(); >> req.oldidx = 0; >> req.newidx = 0; >> error = sysctl_root(0, name, namelen, &req); >> SYSCTL_UNLOCK(); >> } while (error == EAGAIN); >> >> if (req.lock == REQ_WIRED && req.validlen > 0) >> vsunlock(req.oldptr, req.validlen); >> > > Tried no difference unfortunately, possibly its give the closeness > of the lock unlock its just reacquiring the lock straight after releasing > it hence not giving the other thread chance to run / obtain the lock? In an attempt to prove this is in fact the issue I've tried with the locking around this code removed, yes dangerous but just as a test :) It does indeed prevent the extended lockup so this is the right area. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 19:07:22 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D059516A46B; Fri, 1 Feb 2008 19:07:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9455E13C4D5; Fri, 1 Feb 2008 19:07:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 449F448030; Fri, 1 Feb 2008 14:07:21 -0500 (EST) Date: Fri, 1 Feb 2008 19:07:21 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Motin In-Reply-To: <47A2C2A2.5040109@FreeBSD.org> Message-ID: <20080201185435.X88034@fledge.watson.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 19:07:22 -0000 On Fri, 1 Feb 2008, Alexander Motin wrote: > That was actually my second question. As there is only 512 items by default > and they are small in size I can easily preallocate them all on boot. But is > it a good way? Why UMA can't do just the same when I have created zone with > specified element size and maximum number of objects? What is the principal > difference? Alexander, I think we should drill down in the analysis a bit and see if we can figure out what's going on with UMA. What UMA essentially does is ask the VM for pages, and then pack objects into pages. It maintains some meta-data, and depending on the relative sizes of objects and pages, it may store it in the page or potentially elsewhere. Either way, it looks very much an array of struct object. It has a few extra layers of wrapping in order to maintain stats, per-CPU caches, object life cycle, etc. When INVARIANTS is turned off, allocation from the per-CPU cache consists of pulling objects in and out of one of two per-CPU queues. So I guess the question is: where are the cycles going? Are we suffering excessive cache misses in managing the slabs? Are you effectively "cycling through" objects rather than using a smaller set that fits better in the cache? Is some bit of debugging enabled that shouldn't be, perhaps due to a failure of ifdefs? BTW, UMA does let you set the size of buckets, so you can try tuning the bucket size. For starts, try setting the zone flag UMA_ZONE_MAXBUCKET. It would be very helpful if you could try doing some analysis with hwpmc -- "high resolution profiling" is of increasingly limited utility with modern CPUs, where even a high frequency timer won't run very often. It's also quite subject to cycle events that align with other timers in the system. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 20:16:29 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F24916A417; Fri, 1 Feb 2008 20:16:29 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id A430013C459; Fri, 1 Feb 2008 20:16:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71825377; Fri, 01 Feb 2008 22:16:27 +0200 Message-ID: <47A37E14.7050801@FreeBSD.org> Date: Fri, 01 Feb 2008 22:16:20 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> In-Reply-To: <20080201185435.X88034@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 01 Feb 2008 20:20:04 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 20:16:29 -0000 Hi. Robert Watson wrote: > It would be very helpful if you could try doing some analysis with hwpmc > -- "high resolution profiling" is of increasingly limited utility with > modern CPUs, where even a high frequency timer won't run very often. > It's also quite subject to cycle events that align with other timers in > the system. I have tried hwpmc but still not completely friendly with it. Whole picture is somewhat alike to kgmon's, but it looks very noisy. Is there some "know how" about how to use it better? I have tried it for measuring number of instructions. But I am in doubt that instructions is a correct counter for performance measurement as different instructions may have very different execution times depending on many reasons, like cache misses and current memory traffic. I have tried to use tsc to count CPU cycles, but got the error: # pmcstat -n 10000 -S "tsc" -O sample.out pmcstat: ERROR: Cannot allocate system-mode pmc with specification "tsc": Operation not supported What have I missed? I am now using Pentium4 Prescott CPU with HTT enabled in BIOS, but kernel built without SMP to simplify profiling. What counters can you recommend me to use on it for regular time profiling? Thanks for reply. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 21:40:29 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9392E16A417 for ; Fri, 1 Feb 2008 21:40:29 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (pool-96-225-216-68.ptldor.fios.verizon.net [96.225.216.68]) by mx1.freebsd.org (Postfix) with ESMTP id 5ECF513C442 for ; Fri, 1 Feb 2008 21:40:29 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from schitzo.solgatos.com (localhost.home.localnet [127.0.0.1]) by schitzo.solgatos.com (8.14.1/8.13.8) with ESMTP id m11LeSAs013959; Fri, 1 Feb 2008 13:40:28 -0800 Received: from sopwith.solgatos.com (uucp@localhost) by schitzo.solgatos.com (8.14.1/8.13.4/Submit) with UUCP id m11LeSC1013956; Fri, 1 Feb 2008 13:40:28 -0800 Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id VAA00426; Fri, 1 Feb 2008 21:39:15 GMT Message-Id: <200802012139.VAA00426@sopwith.solgatos.com> To: "Steven Hartland" In-reply-to: Your message of "Fri, 01 Feb 2008 13:57:09 GMT." <018701c864da$57020f20$b6db87d4@multiplay.co.uk> Date: Fri, 01 Feb 2008 13:39:15 +0000 From: Dieter Cc: freebsd-performance@freebsd.org Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 21:40:29 -0000 > >> Wait - if it returns EAGAIN for a while, then look at that code above. > >> It will hold the sysctl lock for some indefinite amount of time. Maybe > >> it should look like this instead: > >> > >> > >> do { > >> SYSCTL_LOCK(); > >> req.oldidx = 0; > >> req.newidx = 0; > >> error = sysctl_root(0, name, namelen, &req); > >> SYSCTL_UNLOCK(); > >> } while (error == EAGAIN); > >> > >> if (req.lock == REQ_WIRED && req.validlen > 0) > >> vsunlock(req.oldptr, req.validlen); > >> > > > > Tried no difference unfortunately, possibly its give the closeness > > of the lock unlock its just reacquiring the lock straight after releasing > > it hence not giving the other thread chance to run / obtain the lock? > > In an attempt to prove this is in fact the issue I've tried with the > locking around this code removed, yes dangerous but just as a test :) > It does indeed prevent the extended lockup so this is the right area. Would adding a call to cpu_switch() be appropriate here? From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 22:53:37 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2185716A54B; Fri, 1 Feb 2008 22:53:37 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail04.syd.optusnet.com.au (mail04.syd.optusnet.com.au [211.29.132.185]) by mx1.freebsd.org (Postfix) with ESMTP id 4A1CA13C46A; Fri, 1 Feb 2008 22:53:36 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-219-213.carlnfd3.nsw.optusnet.com.au (c211-30-219-213.carlnfd3.nsw.optusnet.com.au [211.30.219.213]) by mail04.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m11MrV7v010112 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 2 Feb 2008 09:53:32 +1100 Date: Sat, 2 Feb 2008 09:53:31 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Alexander Motin In-Reply-To: <47A37E14.7050801@FreeBSD.org> Message-ID: <20080202080617.U56760@delplex.bde.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A37E14.7050801@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Robert Watson Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 22:53:37 -0000 On Fri, 1 Feb 2008, Alexander Motin wrote: > Robert Watson wrote: >> It would be very helpful if you could try doing some analysis with hwpmc -- >> "high resolution profiling" is of increasingly limited utility with modern You mean "of increasingly greater utility with modern CPUs". Low resolution kernel profiling stopped giving enough resolution in about 1990, and has become of increasingly limited utility since then, but high resolution kernel profiling uses the TSC or possibly a performance counter so it has kept up with CPU speedups. Cache effects and out of order execution are larger now, but they affect all types of profiling and still not too bad with high resulotion kernel profiling. High resolution kernel profiling doesn't really work with SMP, but that is not Alexander's problem since he profiled under UP. >> CPUs, where even a high frequency timer won't run very often. It's also >> quite subject to cycle events that align with other timers in the system. No, it isn't affected by either of these. The TSC timer is incremented on every CPU cycle and the performance counters run are incremented on every event. It is completely unaffected by other timers. > I have tried hwpmc but still not completely friendly with it. Whole picture > is somewhat alike to kgmon's, but it looks very noisy. Is there some "know > how" about how to use it better? hwpmc doesn't work for me either. I can't see how it could work as well as high resolution kernel profiling for events at the single function level, since it is statistics-based. With the statistics clock interrupt rate fairly limited, it just cannot get enough resolution over short runs. Also, it works poorly for me (with a current kernel and ~5.2 userland except for some utilities like pmc*). Generation of profiles stopped working for me, and it often fails with allocation errors. > I have tried it for measuring number of instructions. But I am in doubt that > instructions is a correct counter for performance measurement as different > instructions may have very different execution times depending on many > reasons, like cache misses and current memory traffic. Cycle counts are more useful, but high resolution kernel profiling can do this too, with some fixes: - update perfmon for newer CPUs. It is broken even for Athlons (takes a 2 line fix, or more lines with proper #defines and if()s). - select the performance counter to be used for profiling using sysctl machdep.cputime_clock=$((5 + N)) where N is the number of the performance counter for the instruction count (or any). I use hwpmc mainly to determine N :-). It may also be necessary to change the kernel variable cpu_clock_pmc_conf. Configuration of this is unfinished. - use high resolution kernel profiling normally. Note that switching to a perfmon counter is only permitted of !SMP (since it is too unsupported under SMP to do more than panic if permitted under SMP), and that the switch loses the calibration of profiling. Profiling normally compensates for overheads of the profiling itself, and the compensation would work almoost perfectly for event counters, unlike for time-related counters, since the extra events for profiling aren't much affected by caches. > I have tried to use > tsc to count CPU cycles, but got the error: > # pmcstat -n 10000 -S "tsc" -O sample.out > pmcstat: ERROR: Cannot allocate system-mode pmc with specification "tsc": > Operation not supported > What have I missed? This might be just because the TSC really is not supported. Many things require an APIC for hwpmc to support them. I get errors allocation like this for operations that work a few times before failing. > I am now using Pentium4 Prescott CPU with HTT enabled in BIOS, but kernel > built without SMP to simplify profiling. What counters can you recommend me > to use on it for regular time profiling? Try them all :-). From userland first with an overall count, since looking at the details in gprof output takes too long (and doesn't work for me with hwpmc anyway). I use scripts like the following to try them all from userland: runpm: %%% c="ttcp -n100000 -l5 -u -t epsplex" ctr=0 while test $ctr -lt 256 do ctr1=$(printf "0x%02x\n" $ctr) case $ctr1 in 0x00) src=k8-fp-dispatched-fpu-ops;; 0x01) src=k8-fp-cycles-with-no-fpu-ops-retired;; 0x02) src=k8-fp-dispatched-fpu-fast-flag-ops;; 0x05) src=k8-fp-unknown-$ctr1;; 0x09) src=k8-fp-unknown-$ctr1;; 0x0d) src=k8-fp-unknown-$ctr1;; 0x11) src=k8-fp-unknown-$ctr1;; 0x15) src=k8-fp-unknown-$ctr1;; 0x19) src=k8-fp-unknown-$ctr1;; 0x1d) src=k8-fp-unknown-$ctr1;; 0x20) src=k8-ls-segment-register-load;; # XXX 0x21) src=kx-ls-microarchitectural-resync-by-self-mod-code;; 0x22) src=k8-ls-microarchitectural-resync-by-snoop;; 0x23) src=kx-ls-buffer2-full;; 0x24) src=k8-ls-locked-operation;; # XXX 0x25) src=k8-ls-microarchitectural-late-cancel;; 0x26) src=kx-ls-retired-cflush-instructions;; 0x27) src=kx-ls-retired-cpuid-instructions;; 0x2a) src=kx-ls-unknown-$ctr1;; 0x2b) src=kx-ls-unknown-$ctr1;; 0x2e) src=k7-ls-unknown-$ctr1;; 0x2f) src=k7-ls-unknown-$ctr1;; 0x32) src=kx-ls-unknown-$ctr1;; 0x33) src=kx-ls-unknown-$ctr1;; 0x36) src=k7-ls-unknown-$ctr1;; 0x37) src=k7-ls-unknown-$ctr1;; 0x3a) src=kx-ls-unknown-$ctr1;; 0x3b) src=kx-ls-unknown-$ctr1;; 0x3e) src=k7-ls-unknown-$ctr1;; 0x3f) src=k7-ls-unknown-$ctr1;; 0x40) src=kx-dc-accesses;; 0x41) src=kx-dc-misses;; 0x42) src=kx-dc-refills-from-l2;; # XXX 0x43) src=kx-dc-refills-from-system;; # XXX 0x44) src=kx-dc-writebacks;; # XXX 0x45) src=kx-dc-l1-dtlb-miss-and-l2-dtlb-hits;; 0x46) src=kx-dc-l1-and-l2-dtlb-misses;; 0x47) src=kx-dc-misaligned-references;; 0x48) src=kx-dc-microarchitectural-late-cancel-of-an-access;; 0x49) src=kx-dc-microarchitectural-early-cancel-of-an-access;; 0x4a) src=k8-dc-one-bit-ecc-error;; 0x4b) src=k8-dc-dispatched-prefetch-instructions;; 0x4c) src=k8-dc-dcache-accesses-by-locks;; 0x4d) src=k7-dc-unknown-$ctr1;; 0x4e) src=k7-dc-unknown-$ctr1;; 0x4f) src=k7-dc-unknown-$ctr1;; 0x50) src=kx-dc-unknown-$ctr1;; 0x51) src=kx-dc-unknown-$ctr1;; 0x55) src=kx-dc-unknown-$ctr1;; 0x56) src=kx-dc-unknown-$ctr1;; 0x57) src=kx-dc-unknown-$ctr1;; 0x58) src=kx-dc-unknown-$ctr1;; 0x59) src=kx-dc-unknown-$ctr1;; 0x5d) src=k7-dc-unknown-$ctr1;; 0x5e) src=k7-dc-unknown-$ctr1;; 0x5f) src=k7-dc-unknown-$ctr1;; 0x64) src=kx-bu-unknown-$ctr1;; 0x68) src=kx-bu-unknown-$ctr1;; 0x69) src=kx-bu-unknown-$ctr1;; 0x76) src=kx-bu-cpu-clk-unhalted;; 0x79) src=k8-bu-unknown-$ctr1;; 0x7d) src=k8-bu-internal-l2-request;; # XXX 0x7e) src=k8-bu-fill-request-l2-miss;; # XXX 0x7f) src=k8-bu-fill-into-l2;; # XXX 0x80) src=kx-ic-fetches;; 0x81) src=kx-ic-misses;; 0x82) src=kx-ic-refill-from-l2;; 0x83) src=kx-ic-refill-from-system;; 0x84) src=kx-ic-l1-itlb-misses;; 0x85) src=kx-ic-l1-l2-itlb-misses;; 0x86) src=k8-ic-microarchitectural-resync-by-snoop;; 0x87) src=kx-ic-instruction-fetch-stall;; 0x88) src=kx-ic-return-stack-hit;; 0x89) src=kx-ic-return-stack-overflow;; 0xc0) src=kx-fr-retired-instructions;; 0xc1) src=kx-fr-retired-ops;; 0xc2) src=kx-fr-retired-branches;; 0xc3) src=kx-fr-retired-branches-mispredicted;; 0xc4) src=kx-fr-retired-taken-branches;; 0xc5) src=kx-fr-retired-taken-branches-mispredicted;; 0xc6) src=kx-fr-retired-far-control-transfers;; 0xc7) src=kx-fr-retired-resync-branches;; 0xc8) src=kx-fr-retired-near-returns;; 0xc9) src=kx-fr-retired-near-returns-mispredicted;; 0xca) src=kx-fr-retired-taken-branches-mispred-by-addr-miscompare;; 0xcb) src=k8-fr-retired-fpu-instructions;; 0xcc) src=k8-fr-retired-fastpath-double-op-instructions;; 0xcd) src=kx-fr-interrupts-masked-cycles;; 0xce) src=kx-fr-interrupts-masked-while-pending-cycles;; 0xcf) src=kx-fr-hardware-interrupts;; 0xd0) src=kx-fr-decoder-empty;; 0xd1) src=kx-fr-dispatch-stalls;; 0xd2) src=kx-fr-dispatch-stall-from-branch-abort-to-retire;; 0xd3) src=kx-fr-dispatch-stall-for-serialization;; 0xd4) src=kx-fr-dispatch-stall-for-segment-load;; 0xd5) src=kx-fr-dispatch-stall-when-reorder-buffer-is-full;; 0xd6) src=kx-fr-dispatch-stall-when-reservation-stations-are-full;; 0xd7) src=kx-fr-dispatch-stall-when-fpu-is-full;; 0xd8) src=kx-fr-dispatch-stall-when-ls-is-full;; 0xd9) src=kx-fr-dispatch-stall-when-waiting-for-all-to-be-quiet;; 0xda) src=kx-fr-dispatch-stall-when-far-xfer-or-resync-br-pending;; 0xdb) src=k8-fr-fpu-exceptions;; 0xdc) src=k8-fr-number-of-breakpoints-for-dr0;; 0xdd) src=k8-fr-number-of-breakpoints-for-dr1;; 0xde) src=k8-fr-number-of-breakpoints-for-dr2;; 0xdf) src=k8-fr-number-of-breakpoints-for-dr3;; 0xe0) src=k8-nb-memory-controller-page-access-event;; 0xe1) src=k8-nb-memory-controller-page-table-overflow;; 0xe2) src=k8-nb-memory-controller-dram-slots-missed;; 0xe3) src=k8-nb-memory-controller-turnaround;; 0xe4) src=k8-nb-memory-controller-bypass-saturation;; # XXX 0xe5) src=k8-nb-sized-commands;; # XXX 0xec) src=k8-nb-probe-result;; # XXX 0xf6) src=k8-nb-ht-bus0-bandwidth;; 0xf7) src=k8-nb-ht-bus1-bandwidth;; 0xf8) src=k8-nb-ht-bus2-bandwidth;; 0xfc) src=k8-nb-unknown-$ctr1;; *) src=very-unknown-$ctr1;; esac case $src in k8-*) ctr=$(($ctr + 1)); continue;; *unknown-*) ctr=$(($ctr + 1)); continue;; esac echo "# s/$src " perfmon -c "$c" -ou -l 1 $ctr | egrep -v '(^total: |^mean: |^clocks \(at)' | sed -e 's/1: //' ctr=$(($ctr + 1)) done %%% runpmc: %%% for i in \ k8-fp-dispatched-fpu-ops \ k8-fp-cycles-with-no-fpu-ops-retired \ k8-fp-dispatched-fpu-fast-flag-ops \ k8-ls-segment-register-load \ k8-ls-microarchitectural-resync-by-self-modifying-code \ k8-ls-microarchitectural-resync-by-snoop \ k8-ls-buffer2-full \ k8-ls-locked-operation \ k8-ls-microarchitectural-late-cancel \ k8-ls-retired-cflush-instructions \ k8-ls-retired-cpuid-instructions \ k8-dc-access \ k8-dc-miss \ k8-dc-refill-from-l2 \ k8-dc-refill-from-system \ k8-dc-copyback \ k8-dc-l1-dtlb-miss-and-l2-dtlb-hit \ k8-dc-l1-dtlb-miss-and-l2-dtlb-miss \ k8-dc-misaligned-data-reference \ k8-dc-microarchitectural-late-cancel-of-an-access \ k8-dc-microarchitectural-early-cancel-of-an-access \ k8-dc-one-bit-ecc-error \ k8-dc-dispatched-prefetch-instructions \ k8-dc-dcache-accesses-by-locks \ k8-bu-cpu-clk-unhalted \ k8-bu-internal-l2-request \ k8-bu-fill-request-l2-miss \ k8-bu-fill-into-l2 \ k8-ic-fetch \ k8-ic-miss \ k8-ic-refill-from-l2 \ k8-ic-refill-from-system \ k8-ic-l1-itlb-miss-and-l2-itlb-hit \ k8-ic-l1-itlb-miss-and-l2-itlb-miss \ k8-ic-microarchitectural-resync-by-snoop \ k8-ic-instruction-fetch-stall \ k8-ic-return-stack-hit \ k8-ic-return-stack-overflow \ k8-fr-retired-x86-instructions \ k8-fr-retired-uops \ k8-fr-retired-branches \ k8-fr-retired-branches-mispredicted \ k8-fr-retired-taken-branches \ k8-fr-retired-taken-branches-mispredicted \ k8-fr-retired-far-control-transfers \ k8-fr-retired-resyncs \ k8-fr-retired-near-returns \ k8-fr-retired-near-returns-mispredicted \ k8-fr-retired-taken-branches-mispredicted-by-addr-miscompare \ k8-fr-retired-fpu-instructions \ k8-fr-retired-fastpath-double-op-instructions \ k8-fr-interrupts-masked-cycles \ k8-fr-interrupts-masked-while-pending-cycles \ k8-fr-taken-hardware-interrupts \ k8-fr-decoder-empty \ k8-fr-dispatch-stalls \ k8-fr-dispatch-stall-from-branch-abort-to-retire \ k8-fr-dispatch-stall-for-serialization \ k8-fr-dispatch-stall-for-segment-load \ k8-fr-dispatch-stall-when-reorder-buffer-is-full \ k8-fr-dispatch-stall-when-reservation-stations-are-full \ k8-fr-dispatch-stall-when-fpu-is-full \ k8-fr-dispatch-stall-when-ls-is-full \ k8-fr-dispatch-stall-when-waiting-for-all-to-be-quiet \ k8-fr-dispatch-stall-when-far-xfer-or-resync-branch-pending \ k8-fr-fpu-exceptions \ k8-fr-number-of-breakpoints-for-dr0 \ k8-fr-number-of-breakpoints-for-dr1 \ k8-fr-number-of-breakpoints-for-dr2 \ k8-fr-number-of-breakpoints-for-dr3 \ k8-nb-memory-controller-page-table-overflow \ k8-nb-memory-controller-dram-slots-missed \ k8-nb-memory-controller-bypass-saturation \ k8-nb-sized-commands \ k8-nb-probe-result do pmcstat -s $i sleep 1 2>&1 | sed -e 's/^ *//' -e 's/ */ /' \ -e 's/ *$//' -e 's/\/00\/k8-/\/k8-/' done %%% runpmc7: %%% for i in \ k7-dc-accesses \ k7-dc-misses \ k7-dc-refills-from-l2 \ k7-dc-refills-from-system \ k7-dc-writebacks \ k7-l1-dtlb-miss-and-l2-dtlb-hits \ k7-l1-and-l2-dtlb-misses \ k7-misaligned-references \ k7-ic-fetches \ k7-ic-misses \ k7-l1-itlb-misses \ k7-l1-l2-itlb-misses \ k7-retired-instructions \ k7-retired-ops \ k7-retired-branches \ k7-retired-branches-mispredicted \ k7-retired-taken-branches \ k7-retired-taken-branches-mispredicted \ k7-retired-far-control-transfers \ k7-retired-resync-branches \ k7-interrupts-masked-cycles \ k7-interrupts-masked-while-pending-cycles \ k7-hardware-interrupts do pmcstat -s $i sleep 1 2>&1 | sed -e 's/^ *//' -e 's/ */ /' -e 's/ *$//' -e 's/k7/kx/' done %%% "runpm" tries up to all 256 perfomance counters, with names like the hwpmc ones. k7 means AthlonXP only; k8 means Athlon64 only; kx means both, but many kx's don't really work or are not documented for both. A few like kx-fr-retired-near-returns-mispredicted (?) are not documented for AXP but seem to work and are useful. runpmc tries the documented A64 counters. runpmc7 tries the documented AXP counters. hwpmc is less useful than perfmon here since it doesn't support using the undocumented counters. There is a pmc* option that prints all the counters in the above lists. I checked that they are almost precisely the documented (in Athlon optimization manuals) ones. Names are unfortunately inconsistent between k7 and k8 in some cases, following inconsistencies in the documentation. I don't know anything about Pentium counters except what is in source code. gprof output for the mumble perfmon counter (kx-dc-misses?) while sending 100000 tiny packets using ttcp -t looks like this (after fixing the calibration): %%% granularity: each sample hit covers 16 byte(s) for 0.00% of 2.81 seconds % cumulative self self total time seconds seconds calls us/call us/call name 11.0 0.308 0.308 100083 3 24 syscall [3] 10.8 0.613 0.305 200012 2 2 rn_match [16] 4.4 0.738 0.125 100019 1 1 _bus_dmamap_load_buffer [25] 4.3 0.859 0.121 300107 0 0 generic_copyin [27] 4.0 0.973 0.114 100006 1 9 ip_output [10] 3.8 1.079 0.106 100006 1 4 ether_output [12] 3.7 1.182 0.103 100007 1 1 fgetsock [30] 3.6 1.284 0.102 100006 1 2 bus_dmamap_load_mbuf [21] 3.6 1.385 0.101 200012 1 3 rtalloc_ign [11] 3.6 1.486 0.101 100083 1 25 Xint0x80_syscall [2] 3.6 1.587 0.101 200012 1 1 in_clsroute [32] 3.6 1.688 0.101 100006 1 20 sendto [4] 3.6 1.789 0.101 100008 1 1 in_pcblookup_hash [33] 3.6 1.890 0.101 100006 1 16 kern_sendit [6] 3.6 1.990 0.100 200012 1 2 in_matroute [15] 3.6 2.091 0.100 100748 1 1 doreti [35] 3.6 2.191 0.100 100007 1 2 getsockaddr [22] %%% I would like to be able to do this with hwpmc but don't see how it can. Only (non-statistical) counting at every function call and return can give precise counts like the above. However, non-statistical counting is better for some things. Back to the original problem. Uma allocation overhead shows up in TSC profiles of ttcp, but is just one of too many things that take a while. There are about function calls, each taking about 1%: % granularity: each sample hit covers 16 byte(s) for 0.00% of 0.86 seconds % % % cumulative self self total % time seconds seconds calls ns/call ns/call name % 44.9 0.388 0.388 0 100.00% mcount [1] % 20.9 0.569 0.180 0 100.00% mexitcount [7] % 8.0 0.638 0.069 0 100.00% cputime [14] % 1.8 0.654 0.016 0 100.00% user [25] % 1.6 0.668 0.014 100006 143 1051 udp_output [12] % 1.5 0.681 0.013 100006 133 704 ip_output [13] % 1.3 0.692 0.011 300120 37 37 copyin [30] % 1.2 0.702 0.010 100006 100 1360 sosend_dgram [10] % 0.9 0.710 0.008 200012 39 39 rn_match [38] % 0.9 0.718 0.007 300034 25 25 memcpy [39] % 0.8 0.725 0.007 200103 36 58 uma_zalloc_arg [29] % 0.8 0.732 0.007 100090 68 1977 syscall [4] All the times seem reasonable. Without profiling, sendto() and overheads takes about 1700 nsec in -current and about 1600 nsec in my version of 5.2. (This is for -current. The 100 nsec difference is very hard to understand in detail.) With high resolution kernel profiling, sendto() and overheads take about 8600 nsec here. Profiling has subtracted its own overheads and the result of 1977 nsec for syscall is consistent with syscall() taking a bit less that 1700 nsec when not looked at. (Profiling only subtracts its best-case overheads. Its runtime overheads must be larger due to cache effects, and if these are very large then we cannot trust the compensation. Since it compensated from 8600 down to about 1977, it has clearly down the compensation almost right. The compensation is delicate when there are a lot of functions taking ~20 nsec since the profiling overhead per function call is 82 nsec. % 0.8 0.738 0.007 200012 33 84 rtalloc1 [24] % 0.8 0.745 0.006 100006 65 139 bge_encap [26] % 0.7 0.751 0.006 100006 62 201 bge_start_locked [20] % 0.6 0.757 0.006 200075 28 28 bzero [44] % 0.6 0.761 0.005 100006 48 467 ether_output [15] % 0.6 0.766 0.005 100006 48 192 m_uiotombuf [22] % 0.5 0.771 0.005 200100 23 45 uma_zfree_arg [33] % 0.5 0.775 0.005 100006 46 46 bus_dmamap_load_mbuf_sg [45] % 0.5 0.780 0.004 100028 45 132 malloc [27] % 0.5 0.784 0.004 200012 20 104 rtalloc_ign [19] This is hard to optimize. uma has shown up as taking 58 nsec for uma_zalloc_arg() (including what it calls) and 45 nsec for uma_zfree_arg(). This is on a 2.2GHz A64. Anything larger than that might be a profiling error. But thes allocations here are tiny -- maybe large allocations cause cache methods. I was able to optimize away most the allocation overheads in sendto() be allocating the sockaddr on the stack, but this made little difference overall. (It reduces dynamic allocations per packet from 2 to 1. Both allocations use malloc() so they are a bit slower than pure uma. BTW, switching from mbuf-based allocation to malloc() in getsockaddr() etc. long ago cost 10 usec on a P1/133. A loss of 10000 nsec makes the overhead of 200 nsec for malloc now look tiny.) Remember I said that differences of 100 nsec are hard to understand? It is also not easy to understand why eliminating potentially 100 nsec of malloc() overhead makes almost no difference overall. The 100 nsec gets distributed differently, or maybe the profiling really was wrong for the malloc() part. Reads of the TSC are excuted possibly-out-of-order on some CPUs. This doesn't seem to have much effect on the accuracy of high resolution (TSC) kernel profiling on at least Athlons. rdtsc takes only 12 cycles on AXP-A64. I think it takes much longer on Pentiums. On Phenom it takes ~42 cycles (pessimized to share it across CPUs :-(). With it taking much longer than the functions that it profiles, the compensation might become too fragile. Bruce From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 09:59:45 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4584C16A419; Sat, 2 Feb 2008 09:59:45 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 18B4F13C43E; Sat, 2 Feb 2008 09:59:45 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 760DF4A06C; Sat, 2 Feb 2008 04:59:44 -0500 (EST) Date: Sat, 2 Feb 2008 09:59:44 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Motin In-Reply-To: <47A43873.40801@FreeBSD.org> Message-ID: <20080202095658.R63379@fledge.watson.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 09:59:45 -0000 On Sat, 2 Feb 2008, Alexander Motin wrote: > Robert Watson wrote: >> I guess the question is: where are the cycles going? Are we suffering >> excessive cache misses in managing the slabs? Are you effectively "cycling >> through" objects rather than using a smaller set that fits better in the >> cache? > > In my test setup only several objects from zone usually allocated same time, > but they allocated two times per every packet. > > To check UMA dependency I have made a trivial one-element cache which in my > test case allows to avoid two for four allocations per packet. Avoiding unnecessary allocations is a good general principle, but duplicating cache logic is a bad idea. If you're able to structure the below without using locking, it strikes me you'd do much better, especially if it's in a single processing pass. Can you not use a per-thread/stack/session variable to avoid that? > .....alloc..... > - item = uma_zalloc(ng_qzone, wait | M_ZERO); > + mtx_lock_spin(&itemcachemtx); > + item = itemcache; > + itemcache = NULL; > + mtx_unlock_spin(&itemcachemtx); Why are you using spin locks? They are quite a bit more expensive on several hardwawre platforms, and any environment it's safe to call uma_zalloc() from will be equally safe to use regular mutexes from (i.e., mutex-sleepable). > + if (item == NULL) > + item = uma_zalloc(ng_qzone, wait | M_ZERO); > + else > + bzero(item, sizeof(*item)); > .....free..... > - uma_zfree(ng_qzone, item); > + mtx_lock_spin(&itemcachemtx); > + if (itemcache == NULL) { > + itemcache = item; > + item = NULL; > + } > + mtx_unlock_spin(&itemcachemtx); > + if (item) > + uma_zfree(ng_qzone, item); > ............... > > To be sure that test system is CPU-bound I have throttled it with sysctl to > 1044MHz. With this patch my test PPPoE-to-PPPoE router throughput has grown > from 17 to 21Mbytes/s. Profiling results I have sent promised close results. > >> Is some bit of debugging enabled that shouldn't be, perhaps due to a >> failure of ifdefs? > > I have commented out all INVARIANTS and WITNESS options from GENERIC kernel > config. What else should I check? Hence my request for drilling down a bit on profiling -- the question I'm asking is whether profiling shows things running or taking time that shouldn't be. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 09:31:41 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F1E716A418; Sat, 2 Feb 2008 09:31:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4CCA313C447; Sat, 2 Feb 2008 09:31:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72019456; Sat, 02 Feb 2008 11:31:39 +0200 Message-ID: <47A43873.40801@FreeBSD.org> Date: Sat, 02 Feb 2008 11:31:31 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> In-Reply-To: <20080201185435.X88034@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 02 Feb 2008 12:26:44 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 09:31:41 -0000 Robert Watson wrote: > I guess the question is: where are the cycles going? Are we suffering > excessive cache misses in managing the slabs? Are you effectively > "cycling through" objects rather than using a smaller set that fits > better in the cache? In my test setup only several objects from zone usually allocated same time, but they allocated two times per every packet. To check UMA dependency I have made a trivial one-element cache which in my test case allows to avoid two for four allocations per packet. .....alloc..... - item = uma_zalloc(ng_qzone, wait | M_ZERO); + mtx_lock_spin(&itemcachemtx); + item = itemcache; + itemcache = NULL; + mtx_unlock_spin(&itemcachemtx); + if (item == NULL) + item = uma_zalloc(ng_qzone, wait | M_ZERO); + else + bzero(item, sizeof(*item)); .....free..... - uma_zfree(ng_qzone, item); + mtx_lock_spin(&itemcachemtx); + if (itemcache == NULL) { + itemcache = item; + item = NULL; + } + mtx_unlock_spin(&itemcachemtx); + if (item) + uma_zfree(ng_qzone, item); ............... To be sure that test system is CPU-bound I have throttled it with sysctl to 1044MHz. With this patch my test PPPoE-to-PPPoE router throughput has grown from 17 to 21Mbytes/s. Profiling results I have sent promised close results. > Is some bit of debugging enabled that shouldn't > be, perhaps due to a failure of ifdefs? I have commented out all INVARIANTS and WITNESS options from GENERIC kernel config. What else should I check? -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 10:30:52 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 911C316A417; Sat, 2 Feb 2008 10:30:52 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id CDACC13C46A; Sat, 2 Feb 2008 10:30:51 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72031923; Sat, 02 Feb 2008 12:30:51 +0200 Message-ID: <47A44652.70409@FreeBSD.org> Date: Sat, 02 Feb 2008 12:30:42 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Joseph Koshy References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A37E14.7050801@FreeBSD.org> <84dead720802020209n49c09664p3962fa08f2f9a57c@mail.gmail.com> In-Reply-To: <84dead720802020209n49c09664p3962fa08f2f9a57c@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 02 Feb 2008 12:26:55 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 10:30:52 -0000 Joseph Koshy wrote: > You cannot sample with the TSC since the TSC does not interrupt the CPU. > For CPU cycles you would probably want to use "p4-global-power-events"; > see pmc(3). Thanks, I have already found this. There was only problem, that by default it counts cycles only when both logical cores are active while one of my cores was halted. Sampling on this, profiler shown results close to usual profiling, but looking more random: 175.97 1.49 1/64 ip_input [49] 175.97 1.49 1/64 g_alloc_bio [81] 175.97 1.49 1/64 ng_package_data [18] 1055.81 8.93 6/64 em_handle_rxtx [4] 2639.53 22.32 15/64 em_get_buf [19] 3343.41 28.27 19/64 ng_getqblk [17] 3695.34 31.25 21/64 ip_forward [14] [9]21.6 11262.00 95.23 64 uma_zalloc_arg [9] 35.45 13.03 5/22 critical_exit [75] 26.86 0.00 22/77 critical_enter [99] 19.89 0.00 18/19 mb_ctor_mbuf [141] 31.87 0.24 4/1324 ng_ether_rcvdata [13] 31.87 0.24 4/1324 ip_forward [14] 95.60 0.73 12/1324 ng_iface_rcvdata [16] 103.57 0.79 13/1324 m_freem [25] 876.34 6.71 110/1324 mb_free_ext [30] 9408.75 72.01 1181/1324 ng_free_item [11] [10]20.2 10548.00 80.73 1324 uma_zfree_arg [10] 26.86 0.00 22/77 critical_enter [99] 15.00 11.59 7/7 mb_dtor_mbuf [134] 19.00 6.62 4/4 mb_dtor_pack [136] 1.66 0.00 1/32 m_tag_delete_chain [114] 21.4 11262.00 11262.00 64 175968.75 177456.76 uma_zalloc_arg [9] 20.1 21810.00 10548.00 1324 7966.77 8027.74 uma_zfree_arg [10] 5.6 24773.00 2963.00 1591 1862.35 2640.07 ng_snd_item [15] 3.5 26599.00 1826.00 33 55333.33 55333.33 ng_address_hook [20] 2.4 27834.00 1235.00 319 3871.47 3871.47 ng_acquire_read [28] To make statistics better I need to record sampling data with smaller period, but too much data creates additional overhead including disc operations and brakes statistics. Is there any way to make it more precise? What sampling parameters should I use for better results? -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 10:37:14 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E84E816A417 for ; Sat, 2 Feb 2008 10:37:14 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by mx1.freebsd.org (Postfix) with ESMTP id 75DA013C45B for ; Sat, 2 Feb 2008 10:37:14 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so1397594fgg.35 for ; Sat, 02 Feb 2008 02:37:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=G6cj0Eu0WZ+ahxzGM3wGYEeeVXrlAL5uzCfPYLqTkLM=; b=iBUONsD4sADNtcJgxGg2cBfl4maAmyjEbJoI1VuSf+HFszKJ/JmE8ulLSxNDEUdsLjVkpRleHOO6vAbcr/RtwN2u4EtVDxIH2op6UbmbREvvt4IpXG4mP51NgD9ybHu3wKRFG+oXZLfddDy35bxccoXNeRGr8la79EuovO9QfDE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=jAcBMF3LEq2ptlbPrNY1OL9PHVYzOufIjrW2vF6pT/1gU1yor6vBq6jMrQ121GHyoHOuFsyDT186anl7B4NdItqW4l8UuaecWNzEK6aya/YuE7gHO8gkXhfALYBvNqzVO/E8q+SWYBRf67npLKThKYd3RWG5elCgijWm9RQLn8E= Received: by 10.86.71.1 with SMTP id t1mr4203474fga.33.1201946952701; Sat, 02 Feb 2008 02:09:12 -0800 (PST) Received: by 10.86.90.11 with HTTP; Sat, 2 Feb 2008 02:09:12 -0800 (PST) Message-ID: <84dead720802020209n49c09664p3962fa08f2f9a57c@mail.gmail.com> Date: Sat, 2 Feb 2008 15:39:12 +0530 From: "Joseph Koshy" To: "Alexander Motin" In-Reply-To: <47A37E14.7050801@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A37E14.7050801@FreeBSD.org> X-Mailman-Approved-At: Sat, 02 Feb 2008 12:27:04 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 10:37:15 -0000 > I have tried it for measuring number of instructions. But I am in doubt > that instructions is a correct counter for performance measurement as > different instructions may have very different execution times depending > on many reasons, like cache misses and current memory traffic. I have > tried to use tsc to count CPU cycles, but got the error: > # pmcstat -n 10000 -S "tsc" -O sample.out > pmcstat: ERROR: Cannot allocate system-mode pmc with specification > "tsc": Operation not supported > What have I missed? You cannot sample with the TSC since the TSC does not interrupt the CPU. For CPU cycles you would probably want to use "p4-global-power-events"; see pmc(3). Regards, Koshy From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 10:42:38 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8AAF316A417 for ; Sat, 2 Feb 2008 10:42:38 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id 11E3B13C457 for ; Sat, 2 Feb 2008 10:42:37 +0000 (UTC) (envelope-from joseph.koshy@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so1399145fgg.35 for ; Sat, 02 Feb 2008 02:42:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=+4Defw5yHpWxnUeO7bRWk9zwhV3zzswBBKeZEf2YmjY=; b=lO8E76UxFsckP7VUxKocXGiIDFzaLp2QQImtruZKnhl9e4R0MKaFQ5dFTyOCWo3o/S905q/tYyixWV9QAkrDg35mrz4PRkRJTvlLrI9LOtBMDYoVP+Tiuxq/BgSqyuB4X+uXsM2/f3BUeV4O5f1p+K+LTZa+JZm/VpwB5wfECR0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=JY3Q9EdTkJt2w5D8i7+F5zxu+mVR9KIHggjrvDBww8Caj3RFq9UNpwwRoALqdr30ZPhTzmZAZ3nKnMsHhZs8Q6SOtWR1OrN0/SSi1c7D3T/sFW5SqIb5fV9oCSIwgI/NzSByyEGe11BQuEsejK7c+4TPmIBPDjpmZ4XNgbHurHU= Received: by 10.86.89.4 with SMTP id m4mr4239128fgb.14.1201948956492; Sat, 02 Feb 2008 02:42:36 -0800 (PST) Received: by 10.86.90.11 with HTTP; Sat, 2 Feb 2008 02:42:36 -0800 (PST) Message-ID: <84dead720802020242u4a996a86oa2d045969a4bbdcd@mail.gmail.com> Date: Sat, 2 Feb 2008 16:12:36 +0530 From: "Joseph Koshy" To: "Alexander Motin" In-Reply-To: <47A44652.70409@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A37E14.7050801@FreeBSD.org> <84dead720802020209n49c09664p3962fa08f2f9a57c@mail.gmail.com> <47A44652.70409@FreeBSD.org> X-Mailman-Approved-At: Sat, 02 Feb 2008 12:27:11 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 10:42:38 -0000 > Thanks, I have already found this. There was only problem, that by > default it counts cycles only when both logical cores are active while > one of my cores was halted. Did you try the 'active' event modifier: "p4-global-power-events,active=any"? > Sampling on this, profiler shown results close to usual profiling, but > looking more random: Adding '-fno-omit-frame-pointer' to CFLAGS may help hwpmc to capture callchains better. Regards, Koshy From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 19:56:44 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0B1B16A468; Sat, 2 Feb 2008 19:56:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id E58AB13C442; Sat, 2 Feb 2008 19:56:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72230425; Sat, 02 Feb 2008 21:56:43 +0200 Message-ID: <47A4CAFA.5040708@FreeBSD.org> Date: Sat, 02 Feb 2008 21:56:42 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Peter Jeremy References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202193656.GR35170@server.vk2pj.dyndns.org> In-Reply-To: <20080202193656.GR35170@server.vk2pj.dyndns.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Sat, 02 Feb 2008 20:47:41 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 19:56:44 -0000 Peter Jeremy пишет: > On Sat, Feb 02, 2008 at 11:31:31AM +0200, Alexander Motin wrote: >> To check UMA dependency I have made a trivial one-element cache which in my >> test case allows to avoid two for four allocations per packet. > > You should be able to implement this lockless using atomic(9). I haven't > verified it, but the following should work. I have tried this, but man 9 atomic talks: The atomic_readandclear() functions are not implemented for the types ``char'', ``short'', ``ptr'', ``8'', and ``16'' and do not have any variants with memory barriers at this time. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 20:49:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6853E16A419; Sat, 2 Feb 2008 20:49:16 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id D7B7F13C469; Sat, 2 Feb 2008 20:49:15 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m12KnCS5006593 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 3 Feb 2008 07:49:13 +1100 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.1) with ESMTP id m12KnCDH086893; Sun, 3 Feb 2008 07:49:12 +1100 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m12KnBPl086892; Sun, 3 Feb 2008 07:49:11 +1100 (EST) (envelope-from peter) Date: Sun, 3 Feb 2008 07:49:11 +1100 From: Peter Jeremy To: Alexander Motin Message-ID: <20080202204911.GT35170@server.vk2pj.dyndns.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202193656.GR35170@server.vk2pj.dyndns.org> <47A4CAFA.5040708@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Zi0sgQQBxRFxMTsj" Content-Disposition: inline In-Reply-To: <47A4CAFA.5040708@FreeBSD.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.17 (2007-11-01) X-Mailman-Approved-At: Sat, 02 Feb 2008 21:07:09 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 20:49:16 -0000 --Zi0sgQQBxRFxMTsj Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Feb 02, 2008 at 09:56:42PM +0200, Alexander Motin wrote: >Peter Jeremy ?????: >> On Sat, Feb 02, 2008 at 11:31:31AM +0200, Alexander Motin wrote: >>> To check UMA dependency I have made a trivial one-element cache which i= n=20 >>> my test case allows to avoid two for four allocations per packet. >>=20 >> You should be able to implement this lockless using atomic(9). I haven't >> verified it, but the following should work. > >I have tried this, but man 9 atomic talks: > >The atomic_readandclear() functions are not implemented for the types >``char'', ``short'', ``ptr'', ``8'', and ``16'' and do not have any=20 >variants with memory barriers at this time. Hmmm. This seems to be more a documentation bug than missing code: atomic_readandclear_ptr() seems to be implemented on most architectures (the only one where I can't find it is arm) and is already used in malloc(3). --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --Zi0sgQQBxRFxMTsj Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHpNdH/opHv/APuIcRAjglAJ4xNZrFttUitci+60Q+GyVvdziGrQCaAkxf c2uNtF4tELpQ+7FHoc07LLY= =ZYY7 -----END PGP SIGNATURE----- --Zi0sgQQBxRFxMTsj-- From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 22:05:43 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC57616A419; Sat, 2 Feb 2008 22:05:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id E712413C448; Sat, 2 Feb 2008 22:05:42 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 72286663; Sun, 03 Feb 2008 00:05:42 +0200 Message-ID: <47A4E934.1050207@FreeBSD.org> Date: Sun, 03 Feb 2008 00:05:40 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> In-Reply-To: <20080202095658.R63379@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 02 Feb 2008 22:23:36 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 22:05:44 -0000 Robert Watson wrote: > Hence my request for drilling down a bit on profiling -- the question > I'm asking is whether profiling shows things running or taking time that > shouldn't be. I have not yet understood why does it happend, but hwpmc shows huge amount of "p4-resource-stall"s in UMA functions: % cumulative self self total time seconds seconds calls ms/call ms/call name 45.2 2303.00 2303.00 0 100.00% uma_zfree_arg [1] 41.2 4402.00 2099.00 0 100.00% uma_zalloc_arg [2] 1.4 4472.00 70.00 0 100.00% uma_zone_exhausted_nolock [3] 0.9 4520.00 48.00 0 100.00% ng_snd_item [4] 0.8 4562.00 42.00 0 100.00% __qdivrem [5] 0.8 4603.00 41.00 0 100.00% ether_input [6] 0.6 4633.00 30.00 0 100.00% ng_ppp_prepend [7] Probably it explains why "p4-global-power-events" shows many hits into them % cumulative self self total time seconds seconds calls ms/call ms/call name 20.0 37984.00 37984.00 0 100.00% uma_zfree_arg [1] 17.8 71818.00 33834.00 0 100.00% uma_zalloc_arg [2] 4.0 79483.00 7665.00 0 100.00% ng_snd_item [3] 3.0 85256.00 5773.00 0 100.00% __mcount [4] 2.3 89677.00 4421.00 0 100.00% bcmp [5] 2.2 93853.00 4176.00 0 100.00% generic_bcopy [6] , while "p4-instr-retired" does not. % cumulative self self total time seconds seconds calls ms/call ms/call name 11.1 5351.00 5351.00 0 100.00% ng_apply_item [1] 7.9 9178.00 3827.00 0 100.00% legacy_pcib_alloc_msi [2] 4.1 11182.00 2004.00 0 100.00% init386 [3] 4.0 13108.00 1926.00 0 100.00% rn_match [4] 3.5 14811.00 1703.00 0 100.00% uma_zalloc_arg [5] 2.6 16046.00 1235.00 0 100.00% SHA256_Transform [6] 2.2 17130.00 1084.00 0 100.00% ng_add_hook [7] 2.0 18111.00 981.00 0 100.00% ng_rmhook_self [8] 2.0 19054.00 943.00 0 100.00% em_encap [9] For this moment I have invent two possible explanation. One is that due to UMA's cyclic block allocation order it does not fits CPU caches and another that it is somehow related to critical_exit(), which possibly can cause context switch. Does anybody have better explanation how such small and simple in this part function can cause such results? -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 22:41:59 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2224716A41A; Sat, 2 Feb 2008 22:41:59 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 17D6513C4DB; Sat, 2 Feb 2008 22:41:53 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47A4F1AF.9090306@FreeBSD.org> Date: Sat, 02 Feb 2008 23:41:51 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Alexander Motin References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> In-Reply-To: <47A4E934.1050207@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 22:41:59 -0000 Alexander Motin wrote: > Robert Watson wrote: >> Hence my request for drilling down a bit on profiling -- the question >> I'm asking is whether profiling shows things running or taking time >> that shouldn't be. > > I have not yet understood why does it happend, but hwpmc shows huge > amount of "p4-resource-stall"s in UMA functions: > For this moment I have invent two possible explanation. One is that due > to UMA's cyclic block allocation order it does not fits CPU caches and > another that it is somehow related to critical_exit(), which possibly > can cause context switch. Does anybody have better explanation how such > small and simple in this part function can cause such results? You can look at the raw output from pmcstat, which is a collection of instruction pointers that you can feed to e.g. addr2line to find out exactly where in those functions the events are occurring. This will often help to track down the precise causes. Kris From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 22:51:16 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E5AD16A4AC; Sat, 2 Feb 2008 22:51:16 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 520F713C4FF; Sat, 2 Feb 2008 22:51:16 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id F1D384F81F; Sat, 2 Feb 2008 17:51:12 -0500 (EST) Date: Sat, 2 Feb 2008 22:51:12 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <47A4F1AF.9090306@FreeBSD.org> Message-ID: <20080202224923.T66602@fledge.watson.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, Alexander Motin , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 22:51:16 -0000 On Sat, 2 Feb 2008, Kris Kennaway wrote: > Alexander Motin wrote: >> Robert Watson wrote: >>> Hence my request for drilling down a bit on profiling -- the question I'm >>> asking is whether profiling shows things running or taking time that >>> shouldn't be. >> >> I have not yet understood why does it happend, but hwpmc shows huge amount >> of "p4-resource-stall"s in UMA functions: > >> For this moment I have invent two possible explanation. One is that due to >> UMA's cyclic block allocation order it does not fits CPU caches and another >> that it is somehow related to critical_exit(), which possibly can cause >> context switch. Does anybody have better explanation how such small and >> simple in this part function can cause such results? > > You can look at the raw output from pmcstat, which is a collection of > instruction pointers that you can feed to e.g. addr2line to find out exactly > where in those functions the events are occurring. This will often help to > track down the precise causes. There was, FYI, a report a few years ago that there was a measurable improvement from allocating off the free bucket rather than maintaining separate alloc and free buckets. It sounded good at the time but I was never able to reproduce the benefits in my test environment. Now might be a good time to try to revalidate that. Basically, the goal would be to make the pcpu cache FIFO as much as possible as that maximizes the chances that the newly allocated object already has lines in the cache. It's a fairly trivial tweak to the UMA allocation code. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 23:17:53 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56CB716A421; Sat, 2 Feb 2008 23:17:53 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id F11C313C4E1; Sat, 2 Feb 2008 23:17:52 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 8C46D4B0E3; Sat, 2 Feb 2008 18:17:52 -0500 (EST) Date: Sat, 2 Feb 2008 23:17:52 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Motin In-Reply-To: <47A4F8EE.3070202@FreeBSD.org> Message-ID: <20080202231725.F66602@fledge.watson.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> <47A4F1AF.9090306@FreeBSD.org> <20080202224923.T66602@fledge.watson.org> <47A4F8EE.3070202@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, Kris Kennaway , freebsd-performance@freebsd.org, Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 23:17:53 -0000 On Sun, 3 Feb 2008, Alexander Motin wrote: > Robert Watson wrote: >> Basically, the goal would be to make the pcpu cache FIFO as much as >> possible as that maximizes the chances that the newly allocated object >> already has lines in the cache. > > Why FIFO? I think LIFO (stack) should be better for this goal as the last > freed object has more chances to be still present in cache. Sorry, brain-o -- indeed, as I described, LIFO, rather than as a I wrote. :-) Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Sat Feb 2 23:07:28 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9D7116A46C for ; Sat, 2 Feb 2008 23:07:28 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by mx1.freebsd.org (Postfix) with ESMTP id 79CDF13C469 for ; Sat, 2 Feb 2008 23:07:28 +0000 (UTC) (envelope-from max@love2party.net) Received: from vampire.homelinux.org (dslb-088-067-253-196.pools.arcor-ip.net [88.67.253.196]) by mrelayeu.kundenserver.de (node=mrelayeu3) with ESMTP (Nemesis) id 0MKxQS-1JLRG715eg-0004vV; Sat, 02 Feb 2008 23:54:51 +0100 Received: (qmail 32309 invoked by uid 80); 2 Feb 2008 22:54:50 -0000 Received: from 192.168.4.151 (SquirrelMail authenticated user mlaier) by router.laiers.local with HTTP; Sat, 2 Feb 2008 23:54:50 +0100 (CET) Message-ID: <55964.192.168.4.151.1201992890.squirrel@router.laiers.local> In-Reply-To: <47A4E934.1050207@FreeBSD.org> References: <47A25412.3010301@FreeBSD.org> <47A25A0D.2080508@elischer.org> <47A2C2A2.5040109@FreeBSD.org> <20080201185435.X88034@fledge.watson.org> <47A43873.40801@FreeBSD.org> <20080202095658.R63379@fledge.watson.org> <47A4E934.1050207@FreeBSD.org> Date: Sat, 2 Feb 2008 23:54:50 +0100 (CET) From: "Max Laier" To: "Alexander Motin" User-Agent: SquirrelMail/1.4.13 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Provags-ID: V01U2FsdGVkX1+Ju+/s0QcN+PNjks4a/FauZ8KtJgFwSUmiUQ1 jWzR2439SN947FfjF9XzZKt5wW6ZC+wpI1DkUp2L1eZSx8jYJ3 ZSjOQOD7JYvoDNeV+ZKuA== X-Mailman-Approved-At: Sat, 02 Feb 2008 23:53:18 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , Julian Elischer Subject: Re: Memory allocation performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2008 23:07:29 -0000 Am Sa, 2.02.2008, 23:05, schrieb Alexander Motin: > Robert Watson wrote: >> Hence my request for drilling down a bit on profiling -- the question >> I'm asking is whether profiling shows things running or taking time that >> shouldn't be. > > I have not yet understood why does it happend, but hwpmc shows huge > amount of "p4-resource-stall"s in UMA functions: > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 45.2 2303.00 2303.00 0 100.00% uma_zfree_arg [1] > 41.2 4402.00 2099.00 0 100.00% uma_zalloc_arg [2] > 1.4 4472.00 70.00 0 100.00% > uma_zone_exhausted_nolock [3] > 0.9 4520.00 48.00 0 100.00% ng_snd_item [4] > 0.8 4562.00 42.00 0 100.00% __qdivrem [5] > 0.8 4603.00 41.00 0 100.00% ether_input [6] > 0.6 4633.00 30.00 0 100.00% ng_ppp_prepend [7] > > Probably it explains why "p4-global-power-events" shows many hits into > them > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 20.0 37984.00 37984.00 0 100.00% uma_zfree_arg [1] > 17.8 71818.00 33834.00 0 100.00% uma_zalloc_arg [2] > 4.0 79483.00 7665.00 0 100.00% ng_snd_item [3] > 3.0 85256.00 5773.00 0 100.00% __mcount [4] > 2.3 89677.00 4421.00 0 100.00% bcmp [5] > 2.2 93853.00 4176.00 0 100.00% generic_bcopy [6] > > , while "p4-instr-retired" does not. > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 11.1 5351.00 5351.00 0 100.00% ng_apply_item [1] > 7.9 9178.00 3827.00 0 100.00% > legacy_pcib_alloc_msi [2] > 4.1 11182.00 2004.00 0 100.00% init386 [3] > 4.0 13108.00 1926.00 0 100.00% rn_match [4] > 3.5 14811.00 1703.00 0 100.00% uma_zalloc_arg [5] > 2.6 16046.00 1235.00 0 100.00% SHA256_Transform > [6] > 2.2 17130.00 1084.00 0 100.00% ng_add_hook [7] > 2.0 18111.00 981.00 0 100.00% ng_rmhook_self [8] > 2.0 19054.00 943.00 0 100.00% em_encap [9] > > For this moment I have invent two possible explanation. One is that due > to UMA's cyclic block allocation order it does not fits CPU caches and > another that it is somehow related to critical_exit(), which possibly > can cause context switch. Does anybody have better explanation how such > small and simple in this part function can cause such results? I didn't see bzero accounted for in any of the traces in this thread - makes me wonder if that might mean that it's counted within uma_zalloc? Maybe we are calling it twice by accident? I wasn't quite able to figure out the logic of M_ZERO vs. UMA_ZONE_MALLOC etc. ... just a crazy idea. -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News