From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 09:24:28 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92A8416A418 for ; Sun, 14 Oct 2007 09:24:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0A70913C468 for ; Sun, 14 Oct 2007 09:24:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.10) with ESMTPA id 37545700 for freebsd-hackers@freebsd.org; Sun, 14 Oct 2007 11:24:26 +0300 Message-ID: <4711D237.30808@FreeBSD.org> Date: Sun, 14 Oct 2007 11:24:23 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 CC: freebsd-hackers@freebsd.org References: <1192209799.00813194.1192199401@10.7.7.3> <1192213457.00813222.1192203001@10.7.7.3> <1192242238.00813408.1192231802@10.7.7.3> <1192289079.00813583.1192278003@10.7.7.3> In-Reply-To: <1192289079.00813583.1192278003@10.7.7.3> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: Video memory as swap under FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 09:24:28 -0000 Dag-Erling Smørgrav пишет: > Arne Schwabe writes: >> VIdeo RAM may also not be as stable as your main RAM. I mean nobody if a >> bit flips in video ram. > > That may have been true fifteen years ago, but not today. Have the anybody ever seen ECC video RAM? Video RAM usually works on higher frequencies then main RAM and IMHO it must affect stability. For video RAM some percent of errors is really less important, I have seen it myself with my previous video card until it died completely. -- Alexander Motin From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 09:56:39 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 540FE16A419 for ; Sun, 14 Oct 2007 09:56:39 +0000 (UTC) (envelope-from ap@bnc.net) Received: from mailomat.net (mailomat.net [217.110.117.101]) by mx1.freebsd.org (Postfix) with ESMTP id 6B57213C457 for ; Sun, 14 Oct 2007 09:56:37 +0000 (UTC) (envelope-from ap@bnc.net) X-Mailomat-SpamCatcher-Score: 2 [X] X-Mailomat-Cloudmark-Score: 0.000000 [] Received: from [194.39.192.125] (account bnc-mail@mailrelay.mailomat.net HELO bnc.net) by mailomat.net (CommuniGate Pro SMTP 5.1.9) with ESMTPSA id 30618470; Sun, 14 Oct 2007 11:56:37 +0200 X-SpamCatcher-Score: 2 [X] Received: from [194.39.194.123] (account ap HELO [194.39.194.123]) by bnc.net (CommuniGate Pro SMTP 5.1.12) with ESMTPSA id 2962206; Sun, 14 Oct 2007 11:56:34 +0200 In-Reply-To: <4711D237.30808@FreeBSD.org> References: <1192209799.00813194.1192199401@10.7.7.3> <1192213457.00813222.1192203001@10.7.7.3> <1192242238.00813408.1192231802@10.7.7.3> <1192289079.00813583.1192278003@10.7.7.3> <4711D237.30808@FreeBSD.org> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-4--699774057; protocol="application/pkcs7-signature" Message-Id: From: Achim Patzner Date: Sun, 14 Oct 2007 11:56:30 +0200 To: Alexander Motin X-Mailer: Apple Mail (2.752.3) X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hackers@freebsd.org Subject: Re: Video memory as swap under FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 09:56:39 -0000 --Apple-Mail-4--699774057 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8; delsp=yes; format=flowed Am 14.10.2007 um 10:24 schrieb Alexander Motin: > Dag-Erling Sm=C3=B8rgrav =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> Arne Schwabe writes: >>> VIdeo RAM may also not be as stable as your main RAM. I mean =20 >>> nobody if a >>> bit flips in video ram. >> That may have been true fifteen years ago, but not today. > > Have the anybody ever seen ECC video RAM? Lots. Maybe you're just not old enough but some people can still =20 remember TIGA. You could even use them as separate processor... Achim --Apple-Mail-4--699774057-- From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 10:31:13 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E519D16A418 for ; Sun, 14 Oct 2007 10:31:13 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: from smtpgate4.pacific.net.sg (smtpgate4.pacific.net.sg [203.81.36.24]) by mx1.freebsd.org (Postfix) with SMTP id 014EB13C461 for ; Sun, 14 Oct 2007 10:31:12 +0000 (UTC) (envelope-from oceanare@pacific.net.sg) Received: (qmail 981 invoked from network); 14 Oct 2007 10:04:30 -0000 Received: from bb121-7-30-108.singnet.com.sg (HELO P2120.somewherefaraway.com) (oceanare@121.7.30.108) by smtpgate4.pacific.net.sg with ESMTPA; 14 Oct 2007 10:04:30 -0000 Message-ID: <4711E9A3.1070005@pacific.net.sg> Date: Sun, 14 Oct 2007 18:04:19 +0800 From: Erich Dollansky User-Agent: Thunderbird 2.0.0.6 (X11/20070826) MIME-Version: 1.0 To: Achim Patzner References: <1192209799.00813194.1192199401@10.7.7.3> <1192213457.00813222.1192203001@10.7.7.3> <1192242238.00813408.1192231802@10.7.7.3> <1192289079.00813583.1192278003@10.7.7.3> <4711D237.30808@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Alexander Motin Subject: Re: Video memory as swap under FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 10:31:14 -0000 Hi, Achim Patzner wrote: >=20 > Am 14.10.2007 um 10:24 schrieb Alexander Motin: >=20 >> Dag-Erling Sm=C3=B8rgrav =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>> Arne Schwabe writes: >>>> VIdeo RAM may also not be as stable as your main RAM. I mean nobody = >>>> if a >>>> bit flips in video ram. >>> That may have been true fifteen years ago, but not today. >> >> Have the anybody ever seen ECC video RAM? >=20 > Lots. Maybe you're just not old enough but some people can still rememb= er > TIGA. You could even use them as separate processor... >=20 do you talk about the SPEA fire? Those days, it was even possible to talk to the CEO of a company. Erich From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 09:55:23 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7DAD16A420 for ; Sun, 14 Oct 2007 09:55:23 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 78CEF13C45B for ; Sun, 14 Oct 2007 09:55:23 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A5780E.dip.t-dialin.net [84.165.120.14]) by redbull.bpaserver.net (Postfix) with ESMTP id E22A92E111; Sun, 14 Oct 2007 11:55:17 +0200 (CEST) Received: from deskjail (deskjail.Leidinger.net [192.168.1.109]) by outgoing.leidinger.net (Postfix) with ESMTP id 918965B480D; Sun, 14 Oct 2007 11:54:12 +0200 (CEST) Date: Sun, 14 Oct 2007 11:53:44 +0200 From: Alexander Leidinger To: David Naylor Message-ID: <20071014115344.1c0e813b@deskjail> In-Reply-To: <200710130959.45183.blackdragon@highveldmail.co.za> References: <200710130959.45183.blackdragon@highveldmail.co.za> X-Mailer: Claws Mail 3.0.1 (GTK+ 2.10.14; i686-portbld-freebsd7.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-15.4, required 8, autolearn=not spam, BAYES_00 -15.00, RDNS_DYNAMIC 0.10, SMILEY -0.50) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No X-Mailman-Approved-At: Sun, 14 Oct 2007 11:19:57 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Project Ideas and a question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 09:55:23 -0000 Quoting David Naylor (Sat, 13 Oct 2007 09:59:45 +0200): > I have some project ideas (due to lack of technical skills I can not pursue > them at this time but that is no reason not to share :-). If someone thinks > an idea is a good one could you please add it to the appropriate location > (the volunteer projects page???). My ideas: > > 1) Automatic module loading. Create a discovery system that upon identifying > hardware that a module supports, loads the module. This would probably be a > user-land implementation? > Motivation: Additional ease of use (especially with sound) You don't always want to have every device active. What about a modification to this: Create a discovery system that upon identifying hardware that a module supports, prints out corresponding lines for loader.conf and/or the kernel config. > 2) Automatic kernel customisation. A tool that builds a custom kernel with > all the devices for the current system builtin and with everything not needed > removed. > Motivation: Take the hard work out of building a custom kernel] How do you determine what is not needed? You may want to have some more USB drivers in the tree even when not everything is connected ATM. The same for wlan crypto stuff or other optional things. What about a modification again: Create a tool that prints out all kernel modules which are not in active use currently. A tool can not replace the knowledge someone needs about a particular subsystem of the kernel. Such a tool would allow you to identify things you are not using currently, but you are still in charge to decide if you really want to remove it or not. Bye, Alexander. -- In 1962, you could buy a pair of SHARKSKIN SLACKS, with a "Continental Belt," for $10.99!! http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 20:03:19 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A61B16A417 for ; Sun, 14 Oct 2007 20:03:19 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.freebsd.org (Postfix) with ESMTP id D85B213C47E for ; Sun, 14 Oct 2007 20:03:18 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: by malasada.lava.net (Postfix, from userid 102) id 8F7AF153882; Sun, 14 Oct 2007 09:38:14 -1000 (HST) Date: Sun, 14 Oct 2007 09:38:14 -1000 From: Clifton Royston To: FreeBSD hackers list Message-ID: <20071014193813.GA2677@lava.net> Mail-Followup-To: FreeBSD hackers list Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Subject: Creating install CD with custom ports - how to massage INDEX file? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 20:03:19 -0000 I've been building my own install CDs for a planned multi-server upgrade to 6.2Rp8 and ran into one last stumbling block this week. I understand the process a lot better now than I did a few years back when I was doing it for 4.8, but I'm still having trouble pieceing together how I get my own package set onto the CD in a usable form. I built the release with NO_PORTS=yes, because I'm building the ports from my own CVS tree, which is a tightly pared down subset of the /usr/ports CVS, plus locally written software in ports format. I've ensured that the tree is closed under the dependency operation (to use some math jargon) - essentially that means that my ports subset includes all the dependencies of every port I'm including and all of *its* run/build dependencies in the tree, even if not being built. That allows the dependency graph to be calculated and the INDEX-6 file to be built properly. However, copying the INDEX-6 file and my private packages hierarchy into the CD build area doesn't work; I can read them off the CD post-install but sysinstall doesn't see them. It's not a disaster because I can always put the CD back in after booting and install them then, but it would would be nice to get them all zapped in with the initial install. I think from reading the assorted documentation and people's notes on release-building that the problem is that the INDEX-6 file needs to be massaged into a slightly different format for sysinstall, and I am not clear on the right way to do that for an existing package set, if you're not doing it via the release "build everything" approach and not working off the vanilla ports CVS. To simplify matters further, it all fits on disc 1 - it's not a big file set, so it doesn't need to be split onto multiple disks. I've looked through the many fine manuals here, here, and here: (This last comes tantalizingly close but doesn't get me quite there because it doesn't clarify under what directories or hierarchies these scripts should be installed and run, especially if you have a complete package set built.) This probably boils down to just one "make release" subcommand I need to give at the right stage, after putting my copied ports tree and/or package tree in the right magic place, but I'm not getting it. I'd appreciate a hint from anybody who knows this step. -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 20:37:37 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF73216A417 for ; Sun, 14 Oct 2007 20:37:37 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.freebsd.org (Postfix) with ESMTP id 8844C13C45B for ; Sun, 14 Oct 2007 20:37:37 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: by malasada.lava.net (Postfix, from userid 102) id BA379153882; Sun, 14 Oct 2007 10:37:36 -1000 (HST) Date: Sun, 14 Oct 2007 10:37:36 -1000 From: Clifton Royston To: FreeBSD hackers list Message-ID: <20071014203736.GB2677@lava.net> Mail-Followup-To: FreeBSD hackers list Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Subject: A more tenuously package-related question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 20:37:37 -0000 I used to use pkg_update from the 'pkg_install-devel' toolset to upgrade systems via replacement of binary packages. Its implementation had some minor flaws - it was essentially a perl wrapper for an iterative "pkg_delete -f" followed by "pkg_add -f", which made it problematic to upgrade either the perl or pkg_install packages, for instance - but the core idea was excellent. Despite those flaws it was very useful in maintaining servers via binary packages, because it would reconnect the pkgdb dependencies on the old package version to the new package version. However, it's not part of the current base package tools. Is there any better equivalent tool at the moment, or should I just resuscitate the old "pkg_update"? I browsed through /usr/ports/ports-mgmt, but didn't spot anything which did this seemingly simple and important function. (The "pkg_replace" function *sounds* promising but has almost no information on what it actually does.) -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 14 23:19:18 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AB9216A417 for ; Sun, 14 Oct 2007 23:19:18 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.freebsd.org (Postfix) with ESMTP id 538A413C44B for ; Sun, 14 Oct 2007 23:19:18 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: by malasada.lava.net (Postfix, from userid 102) id B05F0153882; Sun, 14 Oct 2007 13:19:17 -1000 (HST) Date: Sun, 14 Oct 2007 13:19:17 -1000 From: Clifton Royston To: soralx@cydem.org Message-ID: <20071014231917.GB29405@lava.net> Mail-Followup-To: soralx@cydem.org, freebsd-hackers@freebsd.org References: <20071014203736.GB2677@lava.net> <20071014160520.07ad521d@soralx> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071014160520.07ad521d@soralx> User-Agent: Mutt/1.4.2.2i Cc: freebsd-hackers@freebsd.org Subject: Re: A more tenuously package-related question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2007 23:19:18 -0000 On Sun, Oct 14, 2007 at 04:05:20PM -0700, soralx@cydem.org wrote: > > > I used to use pkg_update from the 'pkg_install-devel' toolset to > > upgrade systems via replacement of binary packages. Its > > implementation had some minor flaws - it was essentially a perl > > wrapper for an iterative "pkg_delete -f" followed by "pkg_add -f", > > which made it problematic to upgrade either the perl or pkg_install > > packages, for instance - but the core idea was excellent. Despite > > those flaws it was very useful in maintaining servers via binary > > packages, because it would reconnect the pkgdb dependencies on the > > old package version to the new package version. However, it's not > > part of the current base package tools. > > > > Is there any better equivalent tool at the moment, or should I just > > resuscitate the old "pkg_update"? > > Did you try ports-mgmt/portupgrade? You can run it as `portupgrade -P` > for binary updates. Besides actual 'portupgrade', it has a set of > useful tools, too. But be warned -- the utility is snail-slow. I did look at it, but it appeared that it needed to run off the FreeBSD ports tree, whereas I'm building packages from a separate instance of the ports tree in our own CVS, with local modifications, and then deploying these packages on multiple servers. (This time around, I'm planning to not even install the ports tree on servers other than the build server.) I therefore need to use a utility which can operate using only the dependency information in the pkgdb and embedded in the package files themselves. After posting before, I decided to explore pkg_replace, and it appears that it might be able to do what I want with the right options. -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 00:06:40 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E58BA16A420 for ; Mon, 15 Oct 2007 00:06:40 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from pd2mo1so.prod.shaw.ca (idcmail-mo1so.shaw.ca [24.71.223.10]) by mx1.freebsd.org (Postfix) with ESMTP id A393E13C455 for ; Mon, 15 Oct 2007 00:06:40 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from pd4mr2so.prod.shaw.ca (pd4mr2so-qfe3.prod.shaw.ca [10.0.141.213]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JPX00A1RC4XS460@l-daemon> for freebsd-hackers@freebsd.org; Sun, 14 Oct 2007 17:05:21 -0600 (MDT) Received: from pn2ml3so.prod.shaw.ca ([10.0.121.147]) by pd4mr2so.prod.shaw.ca (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTP id <0JPX009YQC4YV240@pd4mr2so.prod.shaw.ca> for freebsd-hackers@freebsd.org; Sun, 14 Oct 2007 17:05:22 -0600 (MDT) Received: from soralx ([24.87.3.133]) by l-daemon (Sun ONE Messaging Server 6.0 HotFix 1.01 (built Mar 15 2004)) with ESMTP id <0JPX0016JC4XURX0@l-daemon> for freebsd-hackers@freebsd.org; Sun, 14 Oct 2007 17:05:21 -0600 (MDT) Date: Sun, 14 Oct 2007 16:05:20 -0700 From: soralx@cydem.org In-reply-to: <20071014203736.GB2677@lava.net> To: cliftonr@lava.net Message-id: <20071014160520.07ad521d@soralx> MIME-version: 1.0 X-Mailer: Claws Mail 3.0.2 (GTK+ 2.10.14; i386-portbld-freebsd6.2) Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7bit References: <20071014203736.GB2677@lava.net> Cc: freebsd-hackers@freebsd.org Subject: Re: A more tenuously package-related question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 00:06:41 -0000 > I used to use pkg_update from the 'pkg_install-devel' toolset to > upgrade systems via replacement of binary packages. Its > implementation had some minor flaws - it was essentially a perl > wrapper for an iterative "pkg_delete -f" followed by "pkg_add -f", > which made it problematic to upgrade either the perl or pkg_install > packages, for instance - but the core idea was excellent. Despite > those flaws it was very useful in maintaining servers via binary > packages, because it would reconnect the pkgdb dependencies on the > old package version to the new package version. However, it's not > part of the current base package tools. > > Is there any better equivalent tool at the moment, or should I just > resuscitate the old "pkg_update"? Did you try ports-mgmt/portupgrade? You can run it as `portupgrade -P` for binary updates. Besides actual 'portupgrade', it has a set of useful tools, too. But be warned -- the utility is snail-slow. > -- Clifton [SorAlx] ridin' VS1400 From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 00:15:40 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82D3016A46B for ; Mon, 15 Oct 2007 00:15:40 +0000 (UTC) (envelope-from tim1timau@yahoo.com) Received: from web50301.mail.re2.yahoo.com (web50301.mail.re2.yahoo.com [206.190.38.55]) by mx1.freebsd.org (Postfix) with SMTP id 35E7E13C474 for ; Mon, 15 Oct 2007 00:15:40 +0000 (UTC) (envelope-from tim1timau@yahoo.com) Received: (qmail 26626 invoked by uid 60001); 14 Oct 2007 23:48:59 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=KvJIyUZSxg9qB5uxO2WyBNVXKwHzhaT7Hcu3VqlKZh7mTmo8plhfj23FoC4hFy3Z1/zMqq7OsntzxX19D8cBZd1VNymEG3ZZ7TytWSMicJ4mFBqxjdVWResSfyP1nlzwLI7+bDH1K40jBe/hew0yW1rRT4fuJojJe3R3E0F2woY=; X-YMail-OSG: GdABjNcVM1m2OuivJRmt2NjSBTRchgHUJuoBDMlmqSsDtkcx612UW7CCPsEeY1EvTASY1Ehao1G5Mjymzagb4OG_JgY7lVO2_9Y_gXG9VXb6_VvgeIv_sgnrj9Zcbw-- Received: from [203.49.197.51] by web50301.mail.re2.yahoo.com via HTTP; Sun, 14 Oct 2007 16:48:58 PDT Date: Sun, 14 Oct 2007 16:48:58 -0700 (PDT) From: Tim Clewlow To: David Naylor , freebsd-hackers@freebsd.org In-Reply-To: <200710130959.45183.blackdragon@highveldmail.co.za> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <241790.26593.qm@web50301.mail.re2.yahoo.com> Cc: Subject: Re: Project Ideas and a question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 00:15:40 -0000 --- David Naylor wrote: > Hi, > > The question first: init does allow a chroot before booting the system > however > does it allow the first device to be unmounted and use the new chroot as the > root device. If it does how can that be achieved. > > My motivation for this: Allow a USB device or CDROM to boot the system, then > init running of the removable device to execute a script that will allow the > system to be setup (such as configure devices, setup gdbe and geli, etc) then > > when init chroot's it unmounts the removable device and allows that device to > > be physically removed. > > I have some project ideas (due to lack of technical skills I can not pursue > them at this time but that is no reason not to share :-). If someone thinks > an idea is a good one could you please add it to the appropriate location > (the volunteer projects page???). My ideas: > > 1) Automatic module loading. Create a discovery system that upon identifying > > hardware that a module supports, loads the module. This would probably be a > user-land implementation? > Motivation: Additional ease of use (especially with sound) > > 2) Automatic kernel customisation. A tool that builds a custom kernel with > all the devices for the current system builtin and with everything not needed > > removed. > Motivation: Take the hard work out of building a custom kernel] > > 3) If question above is not implemented than add to idea list... > > Feedback is welcome. Thank you for listening to me. > > David Hello David, These all sound like great ideas, however they may have a better chance of being implemented on one of the sibling versions of FreeBSD, ie "Desktop BSD" or "PC BSD". These offerings have already made quite large steps in regard to automating the installation of a BSD system. The general aim is to make it as easy as possible for someone unfamiliar with BSD to install a desktop, ie so it can compete with other desktop operating systems that are already very easy to install. By contrast, the native version of FreeBSD is put to many uses, often not involving a desktop, eg servers, routers, embedded systems. The ability to customize/tune pretty much all of the kernel/world is what makes this possible. And so having an automated installation would have to be unbelievably intelligent, and complicated, in order to cater for all these possibilities. Lastly, keep going with the new ideas, they are always welcome. Perhaps you may want to send them to questions@freebsd.org as this would seem to be more appropriate for initial ideas postings - and more people will see them as well :-) Kind Regards, Tim. ____________________________________________________________________________________ Tonight's top picks. What will you watch tonight? Preview the hottest shows on Yahoo! TV. http://tv.yahoo.com/ From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 15:47:05 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2524416A418; Mon, 15 Oct 2007 15:47:05 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id 3A2B013C45D; Mon, 15 Oct 2007 15:47:02 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.13) with ESMTPA id 194160120; Mon, 15 Oct 2007 18:47:05 +0400 Message-ID: <47137D36.1020305@chistydom.ru> Date: Mon, 15 Oct 2007 18:46:14 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 15 Oct 2007 16:55:00 +0000 Cc: Subject: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 15:47:05 -0000 Hi. I have 3 Dell 2850 with DELL PERC4 SCSI RAID5 6x300GB running lighttpd serving flash video at around 200Mbit/s. %grep amr /var/run/dmesg.boot amr0: mem 0xf80f0000-0xf80fffff,0xfe9c0000-0xfe9fffff irq 46 at device 14.0 on pci2 amr0: Using 64-bit DMA amr0: delete logical drives supported by controller amr0: Firmware 521X, BIOS H430, 256MB RAM amr0: delete logical drives supported by controller amrd0: on amr0 amrd0: 1430400MB (2929459200 sectors) RAID 5 (optimal) Trying to mount root from ufs:/dev/amrd0s1a %uname -a FreeBSD ???.ru 6.2-STABLE FreeBSD 6.2-STABLE #2: Mon Oct 8 16:25:20 MSD 2007 llp@???.ru:/usr/obj/usr/src/sys/SMP-amd64-HWPMC amd64 % After some time of running under high load disk performance become expremely poor. At that periods 'systat -vm 1' shows something like this: Disks amrd0 KB/t 85.39 tps 5 MB/s 0.38 % busy 99 It shows 100% load and just 2-10 tps. There's nothing bad in /var/log/messages or 'netstat -m' or 'vmstat -z' or anywhere else. This continues 15 - 30 minutes or so and everything becomes fine again. After some time - 10 - 12 hours it repeats. Apart of all, I tried to make mutex profiling and here's the results (sorted by the total number of acquisitions): Bad case: 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 (mbuf) 352 160635 173663 0 10896 9678 /usr/src/sys/vm/uma_core.c:2209 (mbuf) 110 134910 173575 0 10838 9464 /usr/src/sys/vm/uma_core.c:2104 (mbuf) 1104 1335319 106888 12 27 1259 /usr/src/sys/netinet/tcp_output.c:253 (so_snd) 171 77754 97685 0 176 207 /usr/src/sys/net/pfil.c:71 (pfil_head_mtx) 140 77104 97685 0 151 128 /usr/src/sys/netinet/ip_fw2.c:164 (IPFW static rules) 100 76543 97685 0 146 45450 /usr/src/sys/netinet/ip_fw2.c:156 (IPFW static rules) 82 77149 97685 0 243 141221 /usr/src/sys/net/pfil.c:63 (pfil_head_mtx) 1644 914481 97679 9 739 949977 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2320 (ipf filter load/unload mutex) 1642 556643 97679 5 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2455 (ipf filter rwlock) 107 89413 97679 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2142 (ipf cache rwlock) 907 148940 81439 1 3 7447 /usr/src/sys/kern/kern_lock.c:168 (lockbuilder mtxpool) 1764 152282 63435 2 438 336763 /usr/src/sys/net/route.c:197 (rtentry) And in the good case: 1738 821795 553033 1 41 284 /usr/src/sys/netinet/tcp_output.c:253 (so_snd) 2770 983643 490815 2 6 54 /usr/src/sys/kern/kern_lock.c:168 (lockbuilder mtxpool) 106 430941 477500 0 5555 4507 /usr/src/sys/netinet/ip_fw2.c:164 (IPFW static rules) 95 423926 477500 0 4412 5645 /usr/src/sys/netinet/ip_fw2.c:156 (IPFW static rules) 94 427239 477500 0 6323 7453 /usr/src/sys/net/pfil.c:63 (pfil_head_mtx) 82 432359 477500 0 5244 5768 /usr/src/sys/net/pfil.c:71 (pfil_head_mtx) 296 4751550 477498 9 20837 23019 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2320 (ipf filter load/unload mutex) 85 2913118 477498 6 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2455 (ipf filter rwlock) 55 473891 477498 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2142 (ipf cache rwlock) 59 291035 309222 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2169 (ipf cache rwlock) 1627 697811 305094 2 2161 2535 /usr/src/sys/net/route.c:147 (radix node head) 232 804172 305094 2 12193 6500 /usr/src/sys/net/route.c:197 (rtentry) 148 892580 303518 2 594 649 /usr/src/sys/net/route.c:1281 (rtentry) 145 584970 303518 1 13479 11148 /usr/src/sys/net/route.c:1265 (rtentry) 121 282669 303518 0 3529 886 /usr/src/sys/net/if_ethersubr.c:409 (em0) Here you can see that high UMA activity happens in periods of low disk performance. But I'm not sure whether this is a root of the problem, not a consequence. I have similiar servers around doing the same things, and they work fine. Also I had the same problem a year ago with another project and that time nothing helped and i had to install Linux. I can provide additional information regarding this server if needed. What else can I try to solve the problem??? With best regards, Alexey Popov From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 17:17:59 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8D2016A417 for ; Mon, 15 Oct 2007 17:17:59 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from el-out-1112.google.com (el-out-1112.google.com [209.85.162.182]) by mx1.freebsd.org (Postfix) with ESMTP id 66BF613C447 for ; Mon, 15 Oct 2007 17:17:59 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by el-out-1112.google.com with SMTP id r27so327339ele for ; Mon, 15 Oct 2007 10:17:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=eWiiF6fBSWheOuOB8Oi+Dgn1RSTuzdzVwoRR0wkasiE=; b=gzi7GbDSyslf5COTKLWDkGs22aEZg6IFJLuzaloE5GEOOvQysEwshhlBZVHwPDFFwYNvjjEac3BCRYTYQX9V4LhD3DCVjv7jV4Oo1aH0dZTZA7/nXYiYi06x+l2yyQKz2NMgm5vkf4JHWJJTxazf4d32gwhNmuQDEEcR9rrEaZA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=Ts+SdWAGf7np9kttiKm5zou/paZXt0GK5QaQbdqkZmOtWZRjxoAK4tcMsNyQdzzmB+Gnsn+e5E7BgLLayTlI1eav0QhwNEFr+kXpJBCfr7N+i5DamAAkz1FbkgRFEa3weDIdM3AoZ9XNb5m7etvev7t6kh90QGQyTl08591hNso= Received: by 10.142.231.7 with SMTP id d7mr1607023wfh.1192468677719; Mon, 15 Oct 2007 10:17:57 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Mon, 15 Oct 2007 10:17:57 -0700 (PDT) Message-ID: <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> Date: Mon, 15 Oct 2007 10:17:57 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org In-Reply-To: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> MIME-Version: 1.0 References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 17:18:00 -0000 Hi, I have an new NFS server that is processing roughly 15mbit of NFS traffic that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, and is serving NFS out a single em nic to LAN clients. The machine works great just serving NFS, but when I try to copy data from one raid volume to another for backups, the machine's NFS performance goes way down, and NFSops start taking multiple seconds to perform. The file copy goes quite quickly, as would be expected. The console of the machine also starts to lag pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do the backup. My first impression was that I was having interrupt issues, since during the backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% CPU processing interrupts). So I recompiled the kernel with polling enabled and enabled it on the NICs. The strange thing is that polling shows enabled in ifconfig, but systat -vm still shows the same amount of interrupts. I get the same performance with polling enabled. I'm looking for some guidance on why the machine bogs so much during what seems to me to be something that should barely impact machine performance at all, and also why polling didn't seem to lower the number of interrupts processed. The old machine was 6 years old running an old intel raid5, and it handled NFS and the concurrent file copies without a sweat. My 3ware is setup as follows: a 2 disk mirror, for the system a 4 disk raid10, for /mnt/data1 a 4 disk raid10, for /mnt/data2 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4125257728 (3934 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf8c00400-0xf8c007ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib13: at device 30.0 on pci0 pci13: on pcib13 pci13: at device 12.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 ata2: on atapci1 ata3: on atapci1 pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2670647184 Hz quality 800 Timecounters tick every 1.000 msec acd0: CDRW at ata0-master UDMA33 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) da1 at twa0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) da2 at twa0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) Trying to mount root from ufs:/dev/da0s1a em1: link state changed to UP [root@madonna ~]# mount /dev/da0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/da0s1e on /usr (ufs, local, soft-updates) /dev/da0s1d on /var (ufs, local, soft-updates) /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates) /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates) [root@madonna ~]# df -hi Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% / devfs 1.0K 1.0K 0B 100% 0 0 100% /dev /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2% /mnt/data1 /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7% /mnt/data2 [root@madonna ~]# more /etc/exports /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah [root@madonna ~]# ifconfig -a em0: flags=8802 mtu 1500 options=4b ether 00:15:17:21:bf:30 media: Ethernet autoselect status: no carrier em1: flags=8843 mtu 1500 options=4b inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x ether 00:15:17:21:bf:31 media: Ethernet 100baseTX status: active lo0: flags=8049 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 Anyone have any clue about what might be going on? --Doug From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 17:28:40 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6786B16A417 for ; Mon, 15 Oct 2007 17:28:40 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from rn-out-0102.google.com (rn-out-0910.google.com [64.233.170.186]) by mx1.freebsd.org (Postfix) with ESMTP id E47C313C481 for ; Mon, 15 Oct 2007 17:28:39 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by rn-out-0102.google.com with SMTP id e5so80441rng for ; Mon, 15 Oct 2007 10:28:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=0AdL9wt3rSAcEziJErCv9dspIf8u71Z4n2ntLhixMvQ=; b=qXGLqdRE3j3Tj5s8B4MgeCKhzbOkwD3APunvnyIdfMj/4lSCn2I+cLe71R9wu9rOtavoBGa6pLp94Z4po/vG/YDVVA0tGIXgqgc3zp3XSsstQblOsXood0cHq5kFt2/G36Wsf2BsMkkqZqlkPNEpS+IVpIl5o4xK5jcDZjtAn1A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=OdK74Ta3rRzjcvr4Lr4mEvM2FK7kpwh7PVkwY2jfXESa2xcXksLuq5upOVjjI9eTndE0PXqRRCdBkstl1RjDYSmSr5FNx5uuAXzIxF1n2lVbbH2kqcl76tRn4DPpB98xz2yMX9eiOMx2mVyPlNh7JIutnWlMT7Jixqq9PHcaWoo= Received: by 10.142.245.10 with SMTP id s10mr1590226wfh.1192467831584; Mon, 15 Oct 2007 10:03:51 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Mon, 15 Oct 2007 10:03:51 -0700 (PDT) Message-ID: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> Date: Mon, 15 Oct 2007 10:03:51 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 17:28:40 -0000 Hi, I have an new NFS server that is processing roughly 15mbit of NFS traffic that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, and is serving NFS out a single em nic to LAN clients. The machine works great just serving NFS, but when I try to copy data from one raid volume to another for backups, the machine's NFS performance goes way down, and NFSops start taking multiple seconds to perform. The file copy goes quite quickly, as would be expected. The console of the machine also starts to lag pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do the backup. My first impression was that I was having interrupt issues, since during the backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% CPU processing interrupts). So I recompiled the kernel with polling enabled and enabled it on the NICs. The strange thing is that polling shows enabled in ifconfig, but systat -vm still shows the same amount of interrupts. I get the same performance with polling enabled. I'm looking for some guidance on why the machine bogs so much during what seems to me to be something that should barely impact machine performance at all, and also why polling didn't seem to lower the number of interrupts processed. The old machine was 6 years old running an old intel raid5, and it handled NFS and the concurrent file copies without a sweat. My 3ware is setup as follows: a 2 disk mirror, for the system a 4 disk raid10, for /mnt/data1 a 4 disk raid10, for /mnt/data2 Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4125257728 (3934 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [GIANT-LOCKED] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf8c00400-0xf8c007ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib13: at device 30.0 on pci0 pci13: on pcib13 pci13: at device 12.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 ata2: on atapci1 ata3: on atapci1 pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2670647184 Hz quality 800 Timecounters tick every 1.000 msec acd0: CDRW at ata0-master UDMA33 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) da1 at twa0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) da2 at twa0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) Trying to mount root from ufs:/dev/da0s1a em1: link state changed to UP [root@madonna ~]# mount /dev/da0s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/da0s1e on /usr (ufs, local, soft-updates) /dev/da0s1d on /var (ufs, local, soft-updates) /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates) /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates) [root@madonna ~]# df -hi Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% / devfs 1.0K 1.0K 0B 100% 0 0 100% /dev /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2% /mnt/data1 /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7% /mnt/data2 [root@madonna ~]# more /etc/exports /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth kelli miho deborah [root@madonna ~]# ifconfig -a em0: flags=8802 mtu 1500 options=4b ether 00:15:17:21:bf:30 media: Ethernet autoselect status: no carrier em1: flags=8843 mtu 1500 options=4b inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x ether 00:15:17:21:bf:31 media: Ethernet 100baseTX status: active lo0: flags=8049 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 Anyone have any clue about what might be going on? --Doug From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 15 17:38:57 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69F5A16A417 for ; Mon, 15 Oct 2007 17:38:57 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (trang.nuxi.org [74.95.12.85]) by mx1.freebsd.org (Postfix) with ESMTP id 3D73013C480 for ; Mon, 15 Oct 2007 17:38:56 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (obrien@localhost [127.0.0.1]) by dragon.nuxi.org (8.14.1/8.14.1) with ESMTP id l9FHcRx0002239; Mon, 15 Oct 2007 10:38:27 -0700 (PDT) (envelope-from obrien@dragon.nuxi.org) Received: (from obrien@localhost) by dragon.nuxi.org (8.14.1/8.14.1/Submit) id l9FHcRq9002238; Mon, 15 Oct 2007 10:38:27 -0700 (PDT) (envelope-from obrien) Date: Mon, 15 Oct 2007 10:38:26 -0700 From: "David O'Brien" To: Yar Tikhiy Message-ID: <20071015173826.GA88628@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Yar Tikhiy , freebsd-hackers@freebsd.org References: <20070901073440.GL85633@comp.chem.msu.su> <46DAFE5C.6070806@freebsd.org> <20070903120353.GH30502@comp.chem.msu.su> <200709261028.43378.jhb@freebsd.org> <20071004022344.GA60878@dragon.NUXI.org> <20071013060138.GA14388@comp.chem.msu.su> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071013060138.GA14388@comp.chem.msu.su> X-Operating-System: FreeBSD 7.0-CURRENT Organization: The NUXI BSD Group X-Pgp-Rsa-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Rsa-Keyid: 1024/34F9F9D5 User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-hackers@freebsd.org Subject: Re: Useful tools missing from /rescue X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 17:38:57 -0000 On Sat, Oct 13, 2007 at 10:01:39AM +0400, Yar Tikhiy wrote: > On Wed, Oct 03, 2007 at 07:23:44PM -0700, David O'Brien wrote: > > I also don't see the need for pgrep - I think needing that says your > > system is running multiuser pretty well. > > First of all, I'd like to point out that /rescue doesn't need to > be as minimal as /stand used to. Now, /rescue is a compact yet > versatile set of essential tools that can help in any difficult > situation when /*bin:/usr/*bin are unusable for some reason, not > only in restoring a broken system while in single-user mode. A .. > As for pgrep+pkill, it can come handy if one has screwed up his > live system and wants to recover it without dropping the system to > single-user. But if we take this just a little bit farther then why don't we go back to a static /[s]bin except for the few things one might need LDAP, etc.. for? That is, what's the purpose in continuing to duplicate /[s]bin into /rescue? /rescue should be just enough to reasonably get a system who's shared libs are messed up working again. /stand was a left-over from installation and not intended to be a sysadmins' savor - it just happened to be because we didn't clean up / after the bits were laid down. > A valid objection to this point is that pgrep's job > can be done with a combination of ps(1) and sed(1), so it's just a > matter of convenience. I guess I'm still having trouble understanding why one would need 'ps' to fix a shared libs issue. Now is a reason to keep adding stuff to /rescue. Also why one would be running 'ps -aux', which is the only way I can think of to get more than one screen of output if a system is in trouble. > The price for it in terms of disk space is next to nothing, and there > are quite useless space hogs in /rescue already (see below on > /rescue/vi.) Considering how few people are skilled in ed(1) these days, we have little choice but include vi. > I won't speak for everyone, but I really like to use fancy shell > commands, particularly during hard times: loops, pipelines, etc. > So I don't have to enter many commands for a single task or browse I guess I'm not creative enough in the ways I've screwed up my systems and needed tools from /rescue. 8-) > > I don't see the purpose of chown - if you have to fall back to /rescue > > you're user 'root' - and you're trying to fix enough so you can use > > standard /*lib & /*bin .. > Having /rescue/chown is just a matter of completeness of the ch* > subset of /rescue tools because chown's job can't be done by any > other stock tools. If /rescue is complete enough, one can find > more applications for it. E.g., the loader, a kernel, and /rescue /rescue wasn't intended to be well orthogonal. /rescue was part of he corner stone of the deal to switch to shared /[s]bin. -- -- David (obrien@FreeBSD.org) From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 00:08:23 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1624816A469 for ; Tue, 16 Oct 2007 00:08:23 +0000 (UTC) (envelope-from myself@rojer.pp.ru) Received: from wooster.rojer.pp.ru (wooster.rojer.pp.ru [80.68.242.188]) by mx1.freebsd.org (Postfix) with ESMTP id 56B1513C46B for ; Tue, 16 Oct 2007 00:08:22 +0000 (UTC) (envelope-from myself@rojer.pp.ru) Received: from wooster.rojer.pp.ru (localhost [127.0.0.1]) by wooster.rojer.pp.ru (Postfix) with ESMTP id 38578117B6 for ; Tue, 16 Oct 2007 03:49:22 +0400 (MSD) X-Spam-Checker-Version: SpamAssassin 3.2.3-rojer (2007-08-08) on wooster.rojer.pp.ru X-Spam-Level: X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3-rojer Received: from nb.rojer.pp.ru (localhost [127.0.0.1]) by wooster.rojer.pp.ru (Postfix) with ESMTP for ; Tue, 16 Oct 2007 03:49:17 +0400 (MSD) Message-ID: <4713FC7D.6070201@rojer.pp.ru> Date: Tue, 16 Oct 2007 00:49:17 +0100 From: Deomid Ryabkov User-Agent: Thunderbird 2.0.0.6 (X11/20070823) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <460D13B0.5070500@rojer.pp.ru> In-Reply-To: <460D13B0.5070500@rojer.pp.ru> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms030203010005010808090603" Subject: Re: 6.2: reproducible hang on amd64, traced to 24h of commits X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 00:08:23 -0000 This is a cryptographically signed message in MIME format. --------------ms030203010005010808090603 Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit fwiw, i have not traced it down to a commit (got fed up with hangs), but conclusively singled out smartmontools as the trigger. after adding 2 more disks, machine wouldn't even boot up past starting smartmontools, locking up hard with the same symptoms. with smartmontools disabled, it booted up and has been up for > 2 months now. Deomid Ryabkov wrote: > ok, now that the machine has been up for 10 days, i am reasonably sure > i've close enough to this one. > > back in january i cvsupped to -STABLE and the box (dual head opteron > box) started hanging. > and i mean it dies completely. > i have all debug options and a working serial console, but still it > just dies and both serial and system console are unresponsive. > no panic message on either, nothing. pretty sad. > > the kernel config is vanilla SMP GENERIC, with all debug options i > could think of enabled (after it started hanging). > > so the first thing i did after rebooting the box a couple of times is > fall back to kernel.old (6.1-STABLE circa august '06). > no hangs. i then started incrementally updating, gradually getting > closer to jan 22. > long story short, i seem to have isolated the problem to commits made > between > date=2006.12.28.00.00.00 and date=2006.12.29.00.00.00. > last hang i had was when running the 12/29 kernel, now it's 12/28 and > the box has been up for 2 weeks already. > based on previois experience i'm pretty certain that this is it. with > bad kernel the box would never stay up more than a few days, never > more than 5. > between 12/28 and 12/29 i see some changes to /sys/amd64/ and > /sys/pci/, which might've be the cause. > i will probably start looking into individual changes, but if anyone > more experienced than me could take a look, it'd be appreciated. > i am willing to try patches. > i confirmed that recent (as of 3 weeks or so) -STABLE still has this > problem. > > thanks in advance. > > ==== > files under /sys that were changed between 12/28 and 12/29: > > Edit src/sys/amd64/amd64/mptable_pci.c > Edit src/sys/amd64/pci/pci_bus.c > Edit src/sys/contrib/dev/ath/public/wackelf.c > Edit src/sys/dev/acpica/acpi_pci.c > Edit src/sys/dev/acpica/acpi_pcib_acpi.c > Edit src/sys/dev/acpica/acpi_pcib_pci.c > Checkout src/sys/dev/ath/if_ath.c > Edit src/sys/dev/cardbus/cardbus.c > Edit src/sys/dev/drm/drm_agpsupport.c > Edit src/sys/dev/pci/pci.c > Edit src/sys/dev/pci/pci_if.m > Edit src/sys/dev/pci/pci_pci.c > Edit src/sys/dev/pci/pci_private.h > Edit src/sys/dev/pci/pcib_private.h > Edit src/sys/dev/pci/pcivar.h > Edit src/sys/i386/i386/mptable_pci.c > Edit src/sys/i386/pci/pci_bus.c > Edit src/sys/kern/subr_bus.c > Checkout src/sys/netgraph/ng_deflate.h > Edit src/sys/pci/agp.c > Edit src/sys/pci/agpreg.h > Edit src/sys/powerpc/ofw/ofw_pcib_pci.c > Edit src/sys/sparc64/pci/apb.c > Edit src/sys/sparc64/pci/ofw_pcib.c > Edit src/sys/sparc64/pci/ofw_pcibus.c > Edit src/sys/sys/param.h > > > ==== > kernel configuration used: > > include GENERIC > > options SMP > > options KDB > options DDB > > makeoptions DEBUG=-g > options INVARIANTS > options INVARIANT_SUPPORT > options WITNESS > options DEBUG_LOCKS > options DEBUG_VFS_LOCKS > options DIAGNOSTIC > ==== > -- Deomid Ryabkov aka Rojer myself@rojer.pp.ru rojer@sysadmins.ru ICQ: 8025844 --------------ms030203010005010808090603 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJPTCC AvkwggJioAMCAQICEBSsKKL5WVjzKP6XqbFuFxowDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUyNzAxMjM1NloX DTA4MDUyNjAxMjM1NlowXzEQMA4GA1UEBBMHUnlhYmtvdjEPMA0GA1UEKhMGRGVvbWlkMRcw FQYDVQQDEw5EZW9taWQgUnlhYmtvdjEhMB8GCSqGSIb3DQEJARYSbXlzZWxmQHJvamVyLnBw LnJ1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArWqlOZVx3IRUSdA6ZnFp2+Su bCUBXwtbtI85NhIm45OugjjzcDoO0bcm2UnYalVzBR9zpRPsUyw53+nWphovBP4adrfCaVHX 9tPE3qDH1sLSuz8RNDwu1joU0w7WLYJIhGjPyv0oWBdEcQJ9HKhCVN9UWZJ9HfYHmXqpNNWF 0iidiVNjAcQs3E+1AK4L9PKryLJxCHRvSiviL9qw843jqfT8B1NJ48W82Tqep0O79CAxWKHY seXwQ294lZxXpNril9bnZ8iVbYhVdFvS3T70mIVP3LrXAjXxIG4vd7n3wsg4uWsOqg/9ChUD Bw/PwwNcLPckEEqL/uFEpmybdjGngwIDAQABoy8wLTAdBgNVHREEFjAUgRJteXNlbGZAcm9q ZXIucHAucnUwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOBgQAX9ky6qWJikV3SSwmF j5wG5rq+svRE+Nv6sIF/OgkABrg9To9iUMjVQV1XjEt5AsdxVJWJFhnAGJXDcfV18QKEwdUz q4RU7aiA4aorOzAXZR+ezF6HZrp0agchh7rcwKJ60EbNZgycrcmPy8UPWjJyn4U6HS4FObr5 q9UB2aHlYDCCAvkwggJioAMCAQICEBSsKKL5WVjzKP6XqbFuFxowDQYJKoZIhvcNAQEFBQAw YjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4x LDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUy NzAxMjM1NloXDTA4MDUyNjAxMjM1NlowXzEQMA4GA1UEBBMHUnlhYmtvdjEPMA0GA1UEKhMG RGVvbWlkMRcwFQYDVQQDEw5EZW9taWQgUnlhYmtvdjEhMB8GCSqGSIb3DQEJARYSbXlzZWxm QHJvamVyLnBwLnJ1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArWqlOZVx3IRU SdA6ZnFp2+SubCUBXwtbtI85NhIm45OugjjzcDoO0bcm2UnYalVzBR9zpRPsUyw53+nWphov BP4adrfCaVHX9tPE3qDH1sLSuz8RNDwu1joU0w7WLYJIhGjPyv0oWBdEcQJ9HKhCVN9UWZJ9 HfYHmXqpNNWF0iidiVNjAcQs3E+1AK4L9PKryLJxCHRvSiviL9qw843jqfT8B1NJ48W82Tqe p0O79CAxWKHYseXwQ294lZxXpNril9bnZ8iVbYhVdFvS3T70mIVP3LrXAjXxIG4vd7n3wsg4 uWsOqg/9ChUDBw/PwwNcLPckEEqL/uFEpmybdjGngwIDAQABoy8wLTAdBgNVHREEFjAUgRJt eXNlbGZAcm9qZXIucHAucnUwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOBgQAX9ky6 qWJikV3SSwmFj5wG5rq+svRE+Nv6sIF/OgkABrg9To9iUMjVQV1XjEt5AsdxVJWJFhnAGJXD cfV18QKEwdUzq4RU7aiA4aorOzAXZR+ezF6HZrp0agchh7rcwKJ60EbNZgycrcmPy8UPWjJy n4U6HS4FObr5q9UB2aHlYDCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJ BgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEa MBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2Vy dmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTEr MCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcw MDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUg Q29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1h aWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065ypla HmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FW y688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEE QB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2 oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3Js MAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxhYmVsMi0x MzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcPf6+svsIXoUOWlJ1/TCG4+DYf qi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH2ydxVyWN3amcOY6MIE9l X5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8xggNkMIIDYAIBATB2 MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQu MSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQFKwoovlZ WPMo/pepsW4XGjAJBgUrDgMCGgUAoIIBwzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwG CSqGSIb3DQEJBTEPFw0wNzEwMTUyMzQ5MTdaMCMGCSqGSIb3DQEJBDEWBBTgCqQSRBTJxGfn 96/hqiYlwza3nDBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIA gDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAE MXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg THRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEBSs KKL5WVjzKP6XqbFuFxowgYcGCyqGSIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMG A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEBSsKKL5WVjzKP6XqbFuFxowDQYJKoZIhvcN AQEBBQAEggEAQ7elxXe9aPA9yKixnt8g1oCLetA4IQt+7auumHqNQxzk3H6thb3S7fl3+Zwi Iw6Jbpm/qunP96NxB3LaOBg9zMnZsQhtu+icig0/M/nh1SZovfAJt27lOcKMW5GcHJrUIiiZ 3z/t1C9leqcH0vcjDlbx49MOesD6eVYSQdHWvxtKYyxLwylRc7PXYv9ZB8nErDozuqmxEcYB o/InlksIPZ8A3wulv2I6fha7PSXQ9nrq/fEx7kH5EdBwY9YnSbr6PDlaZNUt9Q01vqth2MqS QW+QACZTtoRltQ7lVaItCaSyK4PdVaWnRRfjzUqyeUxz+RPja63XEFyCdvEgUz+2OAAAAAAA AA== --------------ms030203010005010808090603-- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 00:42:48 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFAEE16A41B; Tue, 16 Oct 2007 00:42:48 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id 4E91C13C458; Tue, 16 Oct 2007 00:42:47 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47140906.2020107@FreeBSD.org> Date: Tue, 16 Oct 2007 02:42:46 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Alexey Popov References: <47137D36.1020305@chistydom.ru> In-Reply-To: <47137D36.1020305@chistydom.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 00:42:49 -0000 Alexey Popov wrote: > After some time of running under high load disk performance become > expremely poor. At that periods 'systat -vm 1' shows something like this: What does "high load" mean? You need to explain the system workload more. > Disks amrd0 > KB/t 85.39 > tps 5 > MB/s 0.38 > % busy 99 > Apart of all, I tried to make mutex profiling and here's the results > (sorted by the total number of acquisitions): > > Bad case: > > 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) > 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) > 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 (mbuf) > Here you can see that high UMA activity happens in periods of low disk > performance. But I'm not sure whether this is a root of the problem, not > a consequence. The extremely high contention there does seem to say you have a mbuf starvation problem and not a disk problem. I don't know why this would be happening off-hand. Can you also provide more details about the system hardware and configuration? Kris From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 07:43:34 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EC9916A419 for ; Tue, 16 Oct 2007 07:43:34 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mx1.freebsd.org (Postfix) with ESMTP id 0FB4313C458 for ; Tue, 16 Oct 2007 07:43:33 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l9G7hWaM001680 for ; Tue, 16 Oct 2007 17:43:32 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l9G7hWCL001679 for freebsd-hackers@freebsd.org; Tue, 16 Oct 2007 17:43:32 +1000 (EST) (envelope-from peter) Date: Tue, 16 Oct 2007 17:43:32 +1000 From: Peter Jeremy To: FreeBSD hackers list Message-ID: <20071016074331.GJ1184@turion.vk2pj.dyndns.org> References: <20071014193813.GA2677@lava.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dDRMvlgZJXvWKvBx" Content-Disposition: inline In-Reply-To: <20071014193813.GA2677@lava.net> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.16 (2007-06-09) Subject: Re: Creating install CD with custom ports - how to massage INDEX file? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 07:43:34 -0000 --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-Oct-14 09:38:14 -1000, Clifton Royston wrote: > I built the release with NO_PORTS=3Dyes, because I'm building the ports >from my own CVS tree, which is a tightly pared down subset of the >/usr/ports CVS, plus locally written software in ports format. I've >ensured that the tree is closed under the dependency operation (to use >some math jargon) - essentially that means that my ports subset includes >all the dependencies of every port I'm including and all of *its* >run/build dependencies in the tree, even if not being built. That >allows the dependency graph to be calculated and the INDEX-6 file to be >built properly. In which case, you should be able to "cd /usr/ports && make index" --=20 Peter Jeremy --dDRMvlgZJXvWKvBx Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHFGuj/opHv/APuIcRAjjaAJ9lBClYVPv3hx4gdaxn/pSs8XSCzgCfcmlO 20n9KQI+rMym5fXPQz2yIEk= =51u3 -----END PGP SIGNATURE----- --dDRMvlgZJXvWKvBx-- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 09:03:07 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD00916A419; Tue, 16 Oct 2007 09:03:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id EAA1E13C4C5; Tue, 16 Oct 2007 09:03:06 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47147E49.9020301@FreeBSD.org> Date: Tue, 16 Oct 2007 11:03:05 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Alexey Popov References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> In-Reply-To: <47146FB4.6040306@chistydom.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 09:03:07 -0000 Alexey Popov wrote: > Hi. > > Kris Kennaway wrote: > >>> After some time of running under high load disk performance become >>> expremely poor. At that periods 'systat -vm 1' shows something like >>> this: >> What does "high load" mean? You need to explain the system workload >> more. > This web service is similiar to YouTube. This server is video store. I > have around 200G of *.flv (flash video) files on the server. > > I run lighttpd as a web server. Disk load is usually around 50%, network > output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. > > As you can see it is a trivial service - sending files to network via HTTP. A couple of comments. Does lighttpd actually use HTTP accept filters? Are you using ipfilter and ipfw? You are paying a performance penalty for having them. You might try increasing BUCKET_MAX in sys/vm/uma_core.c. I don't really understand the code here, but you seem to be hitting a threshold behaviour where you are constantly running out of space in the per CPU caches. This can happen if your workload is unbalanced between the CPUs and you are always allocating on one but freeing on another, but I wouldn't expect it should happen on your workload. Maybe it can also happen if your turnover is high enough. What does vmstat -z show during the good and bad times? Kris From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 09:41:19 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC37916A418 for ; Tue, 16 Oct 2007 09:41:19 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id 8943A13C467 for ; Tue, 16 Oct 2007 09:41:19 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id A05DD1CE60; Tue, 16 Oct 2007 11:41:18 +0200 (CEST) Date: Tue, 16 Oct 2007 11:41:18 +0200 From: Ed Schouten To: FreeBSD Hackers Message-ID: <20071016094118.GE5411@hoeg.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="dEhLd0jN5MocHkcT" Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) Subject: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: FreeBSD Hackers List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 09:41:19 -0000 --dEhLd0jN5MocHkcT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I asked the following question on questions@, but as requested, I'll forward this question to this list, because of its technical nature. ----- Forwarded message from Ed Schouten ----- > Date: Mon, 15 Oct 2007 23:13:01 +0200 > From: Ed Schouten > To: freebsd-questions@freebsd.org > Subject: Inner workings of turnstiles and sleepqueues >=20 > Hello, >=20 > For some reason, I want to understand how the queueing of blocked > threads in the kernel works when waiting for a lock, which is if I > understand correctly done by the turnstiles and sleepqueues. I'm the > proud owner of The Design and Implementation of the FreeBSD Operating > System book, but for some reason, I can't find anything about it in the > book. >=20 > Is there a way to obtain information about how they work? I already read > the source somewhat, but that shouldn't be an ideal solution, in my > opinion. >=20 ----- End forwarded message ----- Yours, --=20 Ed Schouten WWW: http://g-rave.nl/ --dEhLd0jN5MocHkcT Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHFIc+52SDGA2eCwURAmXnAJwOZl1i+rcy/jj+v98o7XYIObWKTgCfRTf+ 2EDXuaUD95QbMNVijGEgtss= =MsK1 -----END PGP SIGNATURE----- --dEhLd0jN5MocHkcT-- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 10:36:06 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BE8AB16A417 for ; Tue, 16 Oct 2007 10:36:06 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 9646413C468 for ; Tue, 16 Oct 2007 10:36:05 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 4A33E2092; Tue, 16 Oct 2007 12:35:56 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 3D346208D; Tue, 16 Oct 2007 12:35:56 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id 251E784488; Tue, 16 Oct 2007 12:35:56 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: David Naylor References: <200710130959.45183.blackdragon@highveldmail.co.za> Date: Tue, 16 Oct 2007 12:35:56 +0200 In-Reply-To: <200710130959.45183.blackdragon@highveldmail.co.za> (David Naylor's message of "Sat\, 13 Oct 2007 09\:59\:45 +0200") Message-ID: <864pgr73eb.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Project Ideas and a question X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 10:36:06 -0000 David Naylor writes: > 1) Automatic module loading. Create a discovery system that upon > identifying hardware that a module supports, loads the module. This > would probably be a user-land implementation? That is a bad idea. I have a laptop with a broken Ethernet NIC that will trigger an interrupt storm, rendering the system unusable, if you attach a driver to it. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 10:47:35 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 082E116A469 for ; Tue, 16 Oct 2007 10:47:35 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id CB64713C461 for ; Tue, 16 Oct 2007 10:47:34 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id DF2E02094; Tue, 16 Oct 2007 12:47:25 +0200 (CEST) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.0/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id C5E5B2088; Tue, 16 Oct 2007 12:47:25 +0200 (CEST) Received: by ds4.des.no (Postfix, from userid 1001) id A44E384488; Tue, 16 Oct 2007 12:47:25 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Alexander Motin References: <1192209799.00813194.1192199401@10.7.7.3> <1192213457.00813222.1192203001@10.7.7.3> <1192242238.00813408.1192231802@10.7.7.3> <1192289079.00813583.1192278003@10.7.7.3> <4711D237.30808@FreeBSD.org> Date: Tue, 16 Oct 2007 12:47:25 +0200 In-Reply-To: <4711D237.30808@FreeBSD.org> (Alexander Motin's message of "Sun\, 14 Oct 2007 11\:24\:23 +0300") Message-ID: <86zlyj5oaq.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Video memory as swap under FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 10:47:35 -0000 Alexander Motin writes: > Dag-Erling Sm=C3=B8rgrav writes: > > Arne Schwabe writes: > > > Video RAM may also not be as stable as your main RAM. I mean > > > nobody [cares] if a bit flips in video ram. > > That may have been true fifteen years ago, but not today. > Have the anybody ever seen ECC video RAM? Video RAM usually works on > higher frequencies then main RAM and IMHO it must affect stability. > For video RAM some percent of errors is really less important, I have > seen it myself with my previous video card until it died completely. A modern video adapter is basically a processor with a specialized instruction set and multiple parallel pipelines. Video memory is not just a frame buffer, it contains code and structured data (textures, polygon lists etc.) If you're lucky, a single-bit error might just change the color of a pixel, but it might also crash the GPU. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 08:02:34 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6523D16A41B; Tue, 16 Oct 2007 08:02:34 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id EDAD613C459; Tue, 16 Oct 2007 08:02:30 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.13) with ESMTPA id 194309166; Tue, 16 Oct 2007 12:01:44 +0400 Message-ID: <47146FB4.6040306@chistydom.ru> Date: Tue, 16 Oct 2007 12:00:52 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> In-Reply-To: <47140906.2020107@FreeBSD.org> Content-Type: multipart/mixed; boundary="------------040306070507060607050303" X-Mailman-Approved-At: Tue, 16 Oct 2007 11:22:27 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 08:02:34 -0000 This is a multi-part message in MIME format. --------------040306070507060607050303 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi. Kris Kennaway wrote: >> After some time of running under high load disk performance become >> expremely poor. At that periods 'systat -vm 1' shows something like >> this: > What does "high load" mean? You need to explain the system workload more. This web service is similiar to YouTube. This server is video store. I have around 200G of *.flv (flash video) files on the server. I run lighttpd as a web server. Disk load is usually around 50%, network output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. As you can see it is a trivial service - sending files to network via HTTP. > >> Disks amrd0 >> KB/t 85.39 >> tps 5 >> MB/s 0.38 >> % busy 99 > >> Apart of all, I tried to make mutex profiling and here's the results >> (sorted by the total number of acquisitions): >> >> Bad case: >> >> 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) >> 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) >> 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 >> (mbuf) > > > Here you can see that high UMA activity happens in periods of low disk > > performance. But I'm not sure whether this is a root of the problem, not > > a consequence. > > The extremely high contention there does seem to say you have a mbuf > starvation problem and not a disk problem. I don't know why this would > be happening off-hand. But there's no mbuf shortage in `netstat -m`. What else can I try to track down the source of the problem? > Can you also provide more details about the system hardware and > configuration? This is Dell 2850 2 x Xeon 3.2, 4Gb RAM, 6x300Gb SCSI RAID5. I'll attach details. With best regards, Alexey Popov --------------040306070507060607050303 Content-Type: text/plain; name="details.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="details.txt" Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #1: Wed Jul 18 18:12:56 MSD 2007 xxx@xxx.ru:/usr/obj/usr/src/sys/SMP-amd64 module_register: module accf_http already exists! Module accf_http failed to register: 17 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3192.01-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0xf4a Stepping = 10 Features=0xbfebfbff Features2=0x641d AMD Features=0x20100800 AMD Features2=0x1 real memory = 4831838208 (4608 MB) avail memory = 4133355520 (3941 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 ioapic0: Changing APIC ID to 7 ioapic1: Changing APIC ID to 8 ioapic1: WARNING: intbase 32 != expected base 24 ioapic2: Changing APIC ID to 9 ioapic2: WARNING: intbase 64 != expected base 56 ioapic3: Changing APIC ID to 10 ioapic3: WARNING: intbase 96 != expected base 88 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 32-55 on motherboard ioapic2 irqs 64-87 on motherboard ioapic3 irqs 96-119 on motherboard module_register_init: MOD_LOAD (accf_http, 0xffffffff80338880, 0xffffffff807a4720) error 17 acpi0: on motherboard acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 2000 acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 amr0: mem 0xf80f0000-0xf80fffff,0xfe9c0000-0xfe9fffff irq 46 at device 14.0 on pci2 amr0: Using 64-bit DMA amr0: delete logical drives supported by controller amr0: Firmware 521X, BIOS H430, 256MB RAM pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: at device 4.0 on pci0 pci4: on pcib4 pcib5: at device 5.0 on pci0 pci5: on pcib5 pcib6: at device 0.0 on pci5 pci6: on pcib6 em0: port 0xecc0-0xecff mem 0xfe6e0000-0xfe6fffff irq 64 at device 7.0 on pci6 em0: Ethernet address: 00:13:72:62:51:01 pcib7: at device 0.2 on pci5 pci7: on pcib7 em1: port 0xdcc0-0xdcff mem 0xfe4e0000-0xfe4fffff irq 65 at device 8.0 on pci7 em1: Ethernet address: 00:13:72:62:51:02 pcib8: at device 6.0 on pci0 pci8: on pcib8 pcib9: at device 0.0 on pci8 pci9: on pcib9 pcib10: at device 0.2 on pci8 pci10: on pcib10 uhci0: port 0xbce0-0xbcff irq 16 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0xbcc0-0xbcdf irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0xbca0-0xbcbf irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xfeb00000-0xfeb003ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub3: 6 ports with 6 removable, self powered uhub4: vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2 uhub4: multiple transaction translators uhub4: 2 ports with 2 removable, self powered pcib11: at device 30.0 on pci0 pci11: on pcib11 pci11: at device 13.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A orm0: at iomem 0xc0000-0xcafff,0xec000-0xeffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled ipfw2 initialized, divert enabled, rule-based forwarding disabled, default to accept, logging unlimited acd0: CDRW at ata0-master UDMA33 amr0: delete logical drives supported by controller amrd0: on amr0 amrd0: 1430400MB (2929459200 sectors) RAID 5 (optimal) SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/amrd0s1a === machine amd64 cpu HAMMER ident SMP-amd64 # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. options SMP #makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_4BSD # 4BSD scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking #options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories #options QUOTA options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT #options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_IA32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. options IPDIVERT options IPFILTER options IPFILTER_LOG options DUMMYNET options TCP_DROP_SYNFIN options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_DEFAULT_TO_ACCEPT options ACCEPT_FILTER_HTTP options DEVICE_POLLING options COMPAT_LINUX32 # Enable Linux ABI emulation options LINPROCFS # Enable the linux-like proc filesystem support options LINSYSFS # Enable the linux-like sys filesystem support # Bus support. device acpi device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers device ahc # AHA2940 and onboard AIC7xxx devices device ahd # AHA39320/29320 and onboard AIC79xx devices device mpt # LSI-Logic MPT-Fusion # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device pass # Passthrough device (direct SCSI access) # RAID controllers interfaced to the SCSI subsystem device amr # AMI MegaRAID device arcmsr # Areca SATA II RAID #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID #device ciss # Compaq Smart RAID 5* device iir # Intel Integrated RAID #device ips # IBM (Adaptec) ServeRAID device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers #device aac # Adaptec FSA RAID device mfi # LSI MegaRAID SAS device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device kbdmux device vga # VGA video card driver # syscons is the default console driver, resembling an SCO console device sc # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # PCI Ethernet NICs. device em # Intel PRO/1000 adapter Gigabit Ethernet Card # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet device bge # Broadcom BCM570xx Gigabit Ethernet device fxp # Intel EtherExpress PRO/100B (82557, 82558) device re # RealTek 8139C+/8169/8169S/8110S device rl # RealTek 8129/8139 #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device ums # Mouse # Pseudo devices. device loop # Network loopback device random # Entropy device #device mem # Memory and kernel memory devices #device io # I/O device device random # Entropy device device ether # Ethernet support device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device bpf # Berkeley packet filter #device snp # Snoop === KERNCONF= SMP-amd64 CFLAGS= -O2 -pipe NO_BLUETOOTH= true NO_DYNAMICROOT= true NO_FORTRAN= true NO_GAMES= true NO_I4B= true NO_INET6= true NO_LPR= true NO_OBJC= true NO_PROFILE= true NO_SENDMAIL= true NO_SHAREDOCS= true PPP_NO_SUID= true MAKE_IDEA= true BOOTWAIT= 3000 WITHOUT_X11= true # added by use.perl 2007-02-22 16:57:09 PERL_VER=5.8.8 PERL_VERSION=5.8.8 === autoboot_delay="1" kern.ipc.nsfbufs="16384" kern.ipc.nmbclusters="32768" kern.ipc.maxsockets="49152" accf_http_load="YES" === # $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 security.jail.allow_raw_sockets=1 kern.coredump=0 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=32768 net.inet.tcp.blackhole=1 net.inet.udp.blackhole=1 kern.ipc.somaxconn=4096 kern.maxfiles=65535 kern.maxfilesperproc=24576 net.inet.ip.redirect=0 net.inet.icmp.icmplim=100 vfs.ufs.dirhash_maxmem=33554432 === 2271/519/2790 mbufs in use (current/cache/total) 257/301/558/32768 mbuf clusters in use (current/cache/total/max) 257/62 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1081K/731K/1813K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 30771023 requests for I/O initiated by sendfile 22841 calls to protocol drain routines === ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 240, 0, 69, 6, 69, 0 UMA Zones: 376, 0, 69, 1, 69, 0 UMA Slabs: 128, 0, 798, 217, 880024, 0 UMA RCntSlabs: 128, 0, 279, 69, 998108, 0 UMA Hash: 256, 0, 4, 11, 7, 0 16 Bucket: 152, 0, 51, 49, 81, 0 32 Bucket: 280, 0, 16, 54, 67, 5 64 Bucket: 536, 0, 27, 43, 83, 70 128 Bucket: 1048, 0, 248, 91, 4519, 105441 VM OBJECT: 224, 0, 9299, 32929, 39998256, 0 MAP: 352, 0, 7, 15, 7, 0 KMAP ENTRY: 112, 90222, 315, 1467, 8013389, 0 MAP ENTRY: 112, 0, 1540, 935, 107991432, 0 PV ENTRY: 48, 2244600, 22626, 34182, 1346881487, 0 DP fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 1024, 0, 170, 6, 170, 0 16: 16, 0, 3591, 4473, 5890279245, 0 32: 32, 0, 1280, 336, 3464783, 0 64: 64, 0, 6325, 731, 23248175, 0 128: 128, 0, 4932, 491, 13839290, 0 256: 256, 0, 617, 238, 38329528, 0 512: 512, 0, 604, 250, 4068843, 0 1024: 1024, 0, 50, 190, 264828, 0 2048: 2048, 0, 24, 170, 749576, 0 4096: 4096, 0, 246, 153, 2359705, 0 Files: 120, 0, 295, 387, 3661062347, 0 PROC: 856, 0, 89, 71, 1409212, 0 THREAD: 608, 0, 165, 9, 153943, 0 KSEGRP: 136, 0, 161, 73, 161, 0 UPCALL: 88, 0, 3, 73, 3, 0 VMSPACE: 544, 0, 42, 70, 1421441, 0 mbuf_packet: 256, 0, 256, 102, 8363723672, 0 mbuf: 256, 0, 2015, 417, 38877957573, 0 mbuf_cluster: 2048, 32768, 358, 200, 7518256274, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 216, 0, 0, 252, 147835491, 0 ata_request: 336, 0, 0, 22, 24, 0 ata_composite: 376, 0, 0, 0, 0, 0 VNODE: 496, 0, 31168, 19000, 2685396, 0 VNODEPOLL: 152, 0, 0, 50, 1, 0 NAMEI: 1024, 0, 0, 260, 3821583922, 0 S VFS Cache: 104, 0, 30561, 11019, 4538003, 0 L VFS Cache: 327, 0, 225, 315, 37256, 0 NFSMOUNT: 584, 0, 2, 5, 2, 0 NFSNODE: 664, 0, 20, 34, 19966, 0 DIRHASH: 1024, 0, 197, 135, 4270, 0 PIPE: 768, 0, 38, 137, 1054660, 0 KNOTE: 120, 0, 92, 280, 6364470113, 0 socket: 616, 49152, 234, 1680, 10545083, 0 ipq: 56, 1071, 0, 63, 40238, 0 udpcb: 304, 49152, 7, 41, 1228853, 0 inpcb: 304, 49152, 219, 1389, 7473636, 0 tcpcb: 752, 49155, 185, 1365, 7473636, 0 tcptw: 80, 8235, 34, 596, 2262193, 0 syncache: 128, 15370, 0, 145, 7427194, 0 hostcache: 136, 15372, 2277, 271, 1587434, 0 tcpreass: 40, 2100, 0, 420, 247314, 0 sackhole: 32, 0, 14, 491, 217222631, 0 ripcb: 304, 49152, 0, 36, 152, 0 unpcb: 200, 49153, 42, 262, 1842437, 0 rtentry: 264, 0, 7, 35, 37, 0 divcb: 304, 49152, 0, 0, 0, 0 IPFW dynamic rule: 120, 0, 170, 264, 760604, 0 SWAPMETA: 288, 116519, 1, 12, 398, 0 Mountpoints: 792, 0, 10, 5, 11, 0 FFS inode: 192, 0, 31101, 3279, 2665061, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 31101, 2169, 2665061, 0 === Type InUse MemUse HighUse Requests Size(s) PCI Link 16 2K - 16 32,128 acpitask 0 0K - 1 64 cdev 23 6K - 23 256 CAM XPT 6 1K - 9 32,128,512 sigio 1 1K - 1 64 file desc 97 59K - 1411126 16,32,64,128,512,1024,2048,4096 kenv 71 11K - 72 16,32,64 kqueue 2 5K - 1206477 512,2048,4096 proc-args 36 3K - 2116546 16,32,64,128,256 zombie 0 0K - 1409124 256 ithread 144 30K - 144 32,128,256 KTRACE 100 13K - 100 128 linker 51 5K - 113 16,32,64,128,512 lockf 7 1K - 1555267 128 temp 150 443K - 2845070 16,32,64,128,256,512,1024,2048,4096 devbuf 8269 17079K - 1580938 16,32,64,128,256,512,1024,2048,4096 module 204 26K - 205 128 mtx_pool 1 12K - 1 subproc 250 437K - 1409374 512,4096 proc 2 16K - 2 session 26 7K - 314145 256 pgrp 29 4K - 314512 128 acpisem 18 3K - 18 128 cred 94 24K - 27113336 256 uidinfo 5 3K - 140192 64,2048 plimit 21 6K - 1095041 256 sysctltmp 0 0K - 19512 16,32,128 sysctloid 2595 127K - 2595 16,32,64,128 sysctl 0 0K - 2276333 16,32,64 umtx 174 22K - 534 128 SWAP 2 277K - 2 64 CAM dev queue 1 1K - 1 128 bus-sc 58 46K - 1318 16,32,64,128,256,512,1024,2048,4096 bus 611 56K - 3534 16,32,64,128,256,1024 devstat 10 21K - 10 32,4096 eventhandler 49 6K - 49 64,256 CAM queue 3 1K - 3 16 kobj 114 456K - 137 4096 ata_generic 1 1K - 1 1024 rman 112 14K - 563 128 ata_dma 2 1K - 2 256 sbuf 0 0K - 334 16,32,64,128,256,512,1024,2048,4096 sleep queues 175 11K - 535 64 taskqueue 11 2K - 11 16,32,256 turnstiles 175 22K - 535 128 Unitno 11 1K - 307567 32,64 iov 0 0K - 7741292 16,64,128,256,4096 ioctlops 1 1K - 12436597 16,32,64,128,256,512,1024,2048,4096 msg 4 30K - 4 2048,4096 sem 4 8K - 4 1024,2048,4096 shm 1 16K - 1 ttys 1272 180K - 25579 128,1024 ptys 4 1K - 4 256 accf 5 1K - 13 32,64 mbextcnt 2012 32K - 5862464123 16 mbuf_tag 0 0K - 6 32 pcb 18 9K - 670683 16,32,64,128,4096 soname 30 4K - 21014900 16,32,128 acd_driver 1 2K - 1 2048 BIO buffer 0 0K - 72459 1024,2048 vfscache 1 1024K - 1 cluster_save buffer 0 0K - 14419 64,128 VFS hash 1 512K - 1 vnodes 4 1K - 8 32,256 vnodemarker 0 0K - 1634474 512 mount 154 7K - 313 16,32,64,128,256,2048 entropy 1024 64K - 1024 64 BPF 3 1K - 3 128 ether_multi 7 1K - 8 16,64 ifaddr 26 9K - 28 32,64,512,4096 ifnet 4 7K - 4 512,2048 clone 3 12K - 3 4096 arpcom 2 1K - 2 32 lo 1 1K - 1 32 USBdev 21 8K - 21 16,128,256,512 routetbl 144 67K - 276339 32,64,128,256,512 in_multi 2 1K - 2 64 IpFw/IpAcct 20 4K - 60 64,128,2048 hostcache 1 48K - 1 USB 76 13K - 76 16,32,64,128,256,512 syncache 1 12K - 1 NFSV3 diroff 0 0K - 34 512 NFS req 0 0K - 241605 128 NFS daemon 1 16K - 1 p1003.1b 1 1K - 1 16 savedino 0 0K - 44215 256 newdirblk 0 0K - 1672 64 dirrem 0 0K - 3253411 64 mkdir 0 0K - 153290 64 diradd 1 1K - 3275432 64 freefile 0 0K - 2076356 64 freeblks 2 1K - 2377275 256 freefrag 2 1K - 162141 64 allocindir 1 1K - 2401831 128 indirdep 1 1K - 119555 64 allocdirect 4 1K - 3102764 256 bmsafemap 4 1K - 134621 128 newblk 1 1K - 5504596 64,512 inodedep 5 513K - 2273914 256 pagedep 2 129K - 93330 128 UFS dirhash 213 39K - 5703 16,32,64,128,512 UFS mount 18 77K - 21 512,2048,4096 UMAHash 3 10K - 9 512,1024,2048,4096 DEVFS1 94 47K - 94 512 DEVFS3 104 26K - 105 256 DEVFS 11 1K - 12 16,256 VM pgdata 2 129K - 2 128 pfs_nodes 165 21K - 165 128 pfs_vncache 1 1K - 169 64 CAM SIM 1 1K - 1 128 memdesc 1 4K - 1 4096 GEOM 98 23K - 567 16,32,64,128,256,512,1024,2048,4096 isadev 8 1K - 8 128 I/O APIC 4 8K - 4 2048 nexusdev 2 1K - 2 16 CAM periph 1 1K - 1 256 acpidev 69 5K - 69 64 atkbddev 1 1K - 1 64 acpica 2245 242K - 19682 16,32,64,128,256,512,1024,4096 linux 15 1K - 15 64 === last pid: 11008; load averages: 0.07, 0.10, 0.08 up 47+08:32:50 11:46:15 38 processes: 1 running, 37 sleeping Mem: 46M Active, 3443M Inact, 246M Wired, 144M Cache, 208M Buf, 5596K Free Swap: 2048M Total, 4K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56386 root 1 4 0 19856K 10000K kqread 1 115:19 2.88% lighttpd 636 root 1 96 0 18292K 4212K select 0 25:39 0.00% snmpd 784 root 1 96 0 19668K 2072K select 1 2:31 0.00% sshd 680 root 1 96 0 7732K 1384K select 0 1:59 0.00% ntpd 1540 root 1 96 0 35092K 6496K select 0 1:30 0.00% httpd 769 root 4 20 0 14148K 2632K kserel 0 1:04 0.00% bacula-fd 755 root 1 96 0 3852K 1060K select 1 0:22 0.00% master 568 root 1 96 0 3648K 908K select 0 0:18 0.00% syslogd 80663 root 1 8 0 3688K 1016K nanslp 1 0:05 0.00% cron 760 postfix 1 96 0 3944K 1160K select 0 0:04 0.00% qmgr 89776 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89763 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89774 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89775 www 1 96 0 35180K 6684K select 0 0:04 0.00% httpd 699 root 1 20 0 7732K 1388K pause 0 0:03 0.00% ntpd 484 root 1 4 0 652K 220K select 0 0:00 0.00% devd 10904 llp 1 96 0 30616K 3564K select 0 0:00 0.00% sshd 10915 root 1 20 0 3912K 2340K pause 1 0:00 0.00% csh === kern.ostype: FreeBSD kern.osrelease: 6.2-STABLE kern.osrevision: 199506 kern.version: FreeBSD 6.2-STABLE #1: Wed Jul 18 18:12:56 MSD 2007 xxx@xxx.ru:/usr/obj/usr/src/sys/SMP-amd64 kern.maxvnodes: 100000 kern.maxproc: 6164 kern.maxfiles: 65535 kern.argmax: 262144 kern.securelevel: -1 kern.hostname: xxx.ru kern.hostid: 0 kern.clockrate: { hz = 1000, tick = 1000, profhz = 666, stathz = 133 } kern.posix1version: 200112 kern.ngroups: 16 kern.job_control: 1 kern.saved_ids: 0 kern.boottime: { sec = 1188429235, usec = 814920 } Thu Aug 30 03:13:55 2007 kern.domainname: kern.osreldate: 602111 kern.bootfile: /boot/kernel.old/kernel kern.maxfilesperproc: 24576 kern.maxprocperuid: 5547 kern.ipc.maxsockbuf: 262144 kern.ipc.sockbuf_waste_factor: 8 kern.ipc.somaxconn: 4096 kern.ipc.max_linkhdr: 16 kern.ipc.max_protohdr: 40 kern.ipc.max_hdr: 56 kern.ipc.max_datalen: 120 kern.ipc.nmbjumbo16: 0 kern.ipc.nmbjumbo9: 0 kern.ipc.nmbjumbop: 0 kern.ipc.nmbclusters: 32768 kern.ipc.piperesizeallowed: 1 kern.ipc.piperesizefail: 0 kern.ipc.pipeallocfail: 0 kern.ipc.pipefragretry: 0 kern.ipc.pipekva: 573440 kern.ipc.pipes: 70 kern.ipc.maxpipekva: 20971520 kern.ipc.msgseg: 2048 kern.ipc.msgssz: 8 kern.ipc.msgtql: 40 kern.ipc.msgmnb: 2048 kern.ipc.msgmni: 40 kern.ipc.msgmax: 16384 kern.ipc.semaem: 16384 kern.ipc.semvmx: 32767 kern.ipc.semusz: 104 kern.ipc.semume: 10 kern.ipc.semopm: 100 kern.ipc.semmsl: 60 kern.ipc.semmnu: 30 kern.ipc.semmns: 60 kern.ipc.semmni: 10 kern.ipc.semmap: 30 kern.ipc.shm_allow_removed: 0 kern.ipc.shm_use_phys: 0 kern.ipc.shmall: 8192 kern.ipc.shmseg: 128 kern.ipc.shmmni: 192 kern.ipc.shmmin: 1 kern.ipc.shmmax: 33554432 kern.ipc.numopensockets: 242 kern.ipc.maxsockets: 49152 kern.ipc.nsfbufsused: 0 kern.ipc.nsfbufspeak: 0 kern.ipc.nsfbufs: 0 kern.dummy: 0 kern.ps_strings: 140737488355296 kern.usrstack: 140737488355328 kern.logsigexit: 1 kern.iov_max: 1024 kern.cam.cam_srch_hi: 0 kern.cam.scsi_delay: 5000 kern.cam.da.da_send_ordered: 1 kern.cam.da.default_timeout: 60 kern.cam.da.retry_count: 4 kern.disks: amrd0 kern.geom.collectstats: 1 kern.geom.debugflags: 0 kern.elf64.fallback_brand: -1 kern.init_shutdown_timeout: 120 kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init:/stand/sysinstall kern.acct_suspended: 0 kern.acct_chkfreq: 15 kern.acct_resume: 4 kern.acct_suspend: 2 kern.cp_time: 3062137 254 19559404 40888455 1023677682 kern.openfiles: 295 kern.kq_calloutmax: 4096 kern.ps_arg_cache_limit: 256 kern.stackprot: 7 kern.randompid: 0 kern.lastpid: 11060 kern.ktrace.request_pool: 100 kern.ktrace.genio_size: 4096 kern.module_path: /boot/kernel;/boot/modules kern.malloc_count: 170 kern.malloc: Type InUse MemUse HighUse Requests Size(s) linux 15 1K - 15 64 acpica 2245 242K - 20942 16,32,64,128,256,512,1024,4096 atkbddev 1 1K - 1 64 acpidev 69 5K - 69 64 CAM periph 1 1K - 1 256 nexusdev 2 1K - 2 16 I/O APIC 4 8K - 4 2048 isadev 8 1K - 8 128 GEOM 98 23K - 599 16,32,64,128,256,512,1024,2048,4096 memdesc 1 4K - 1 4096 CAM SIM 1 1K - 1 128 pfs_vncache 1 1K - 169 64 pfs_nodes 165 21K - 165 128 VM pgdata 2 129K - 2 128 DEVFS 11 1K - 12 16,256 DEVFS3 104 26K - 105 256 DEVFS1 94 47K - 94 512 UMAHash 3 10K - 9 512,1024,2048,4096 UFS mount 18 77K - 21 512,2048,4096 UFS dirhash 216 40K - 5706 16,32,64,128,512 pagedep 1 128K - 93336 128 inodedep 3 513K - 2273934 256 newblk 1 1K - 5505148 64,512 bmsafemap 2 1K - 134644 128 allocdirect 1 1K - 3102854 256 indirdep 1 1K - 119569 64 allocindir 1 1K - 2402293 128 freefrag 0 0K - 162148 64 freeblks 0 0K - 2377280 256 freefile 0 0K - 2076360 64 diradd 0 0K - 3275441 64 mkdir 0 0K - 153292 64 dirrem 0 0K - 3253416 64 newdirblk 0 0K - 1672 64 savedino 0 0K - 44222 256 p1003.1b 1 1K - 1 16 NFS daemon 1 16K - 1 NFS req 0 0K - 241605 128 NFSV3 diroff 0 0K - 34 512 syncache 1 12K - 1 USB 76 13K - 76 16,32,64,128,256,512 hostcache 1 48K - 1 IpFw/IpAcct 20 4K - 60 64,128,2048 in_multi 2 1K - 2 64 routetbl 144 67K - 276361 32,64,128,256,512 USBdev 21 8K - 21 16,128,256,512 lo 1 1K - 1 32 arpcom 2 1K - 2 32 clone 3 12K - 3 4096 ifnet 4 7K - 4 512,2048 ifaddr 26 9K - 28 32,64,512,4096 ether_multi 7 1K - 8 16,64 BPF 3 1K - 3 128 entropy 1024 64K - 1024 64 mount 154 7K - 313 16,32,64,128,256,2048 vnodemarker 0 0K - 1634619 512 vnodes 4 1K - 8 32,256 VFS hash 1 512K - 1 cluster_save buffer 0 0K - 14420 64,128 vfscache 1 1024K - 1 BIO buffer 5 10K - 72464 1024,2048 acd_driver 1 2K - 1 2048 soname 30 4K - 21016379 16,32,128 pcb 18 9K - 670693 16,32,64,128,4096 mbuf_tag 0 0K - 6 32 mbextcnt 2084 33K -5863097811 16 accf 5 1K - 13 32,64 ptys 4 1K - 4 256 ttys 1272 180K - 25579 128,1024 shm 1 16K - 1 sem 4 8K - 4 1024,2048,4096 msg 4 30K - 4 2048,4096 ioctlops 0 0K - 12439705 16,32,64,128,256,512,1024,2048,4096 iov 0 0K - 7742097 16,64,128,256,4096 Unitno 11 1K - 307567 32,64 turnstiles 175 22K - 535 128 taskqueue 11 2K - 11 16,32,256 sleep queues 175 11K - 535 64 sbuf 0 0K - 572 16,32,64,128,256,512,1024,2048,4096 ata_dma 2 1K - 2 256 rman 112 14K - 563 128 ata_generic 1 1K - 1 1024 kobj 114 456K - 137 4096 CAM queue 3 1K - 3 16 eventhandler 49 6K - 49 64,256 devstat 10 21K - 10 32,4096 bus 611 56K - 4458 16,32,64,128,256,1024 bus-sc 58 46K - 1318 16,32,64,128,256,512,1024,2048,4096 CAM dev queue 1 1K - 1 128 SWAP 2 277K - 2 64 umtx 174 22K - 534 128 sysctl 0 0K - 2276639 16,32,64 sysctloid 2595 127K - 2595 16,32,64,128 sysctltmp 0 0K - 19768 16,32,128 plimit 20 5K - 1095118 256 uidinfo 5 3K - 140199 64,2048 cred 93 24K - 27114414 256 acpisem 18 3K - 18 128 pgrp 28 4K - 314540 128 session 25 7K - 314158 256 proc 2 16K - 2 subproc 245 417K - 1409454 512,4096 mtx_pool 1 12K - 1 module 204 26K - 205 128 devbuf 8267 17079K - 1581116 16,32,64,128,256,512,1024,2048,4096 temp 150 291K - 2845308 16,32,64,128,256,512,1024,2048,4096 lockf 7 1K - 1555519 128 linker 51 5K - 113 16,32,64,128,512 KTRACE 100 13K - 100 128 ithread 144 30K - 144 32,128,256 zombie 0 0K - 1409209 256 proc-args 31 2K - 2116635 16,32,64,128,256 kqueue 2 5K - 1206487 512,2048,4096 kenv 71 11K - 72 16,32,64 file desc 92 57K - 1411206 16,32,64,128,512,1024,2048,4096 sigio 1 1K - 1 64 CAM XPT 6 1K - 9 32,128,512 cdev 23 6K - 23 256 acpitask 0 0K - 1 64 PCI Link 16 2K - 16 32,128 kern.fallback_elf_brand: -1 kern.maxusers: 384 kern.ident: SMP-amd64 kern.polling.idlepoll_sleeping: 1 kern.polling.stalled: 0 kern.polling.suspect: 0 kern.polling.phase: 0 kern.polling.enable: 0 kern.polling.handlers: 0 kern.polling.residual_burst: 0 kern.polling.pending_polls: 0 kern.polling.lost_polls: 0 kern.polling.short_ticks: 0 kern.polling.reg_frac: 20 kern.polling.user_frac: 50 kern.polling.idle_poll: 0 kern.polling.each_burst: 5 kern.polling.burst_max: 150 kern.polling.burst: 5 kern.kstack_pages: 4 kern.shutdown.kproc_shutdown_wait: 60 kern.shutdown.poweroff_delay: 5000 kern.sync_on_panic: 0 kern.corefile: %N.core kern.nodump_coredump: 0 kern.coredump: 0 kern.sugid_coredump: 0 kern.fscale: 2048 kern.timecounter.tick: 1 kern.timecounter.choice: TSC(-100) ACPI-fast(1000) HPET(2000) i8254(0) dummy(-1000000) kern.timecounter.hardware: HPET kern.timecounter.nsetclock: 3 kern.timecounter.ngetmicrotime: 408535457 kern.timecounter.ngetnanotime: 0 kern.timecounter.ngetbintime: 0 kern.timecounter.ngetmicrouptime: 1191179415 kern.timecounter.ngetnanouptime: 1188431 kern.timecounter.ngetbinuptime: 2665117 kern.timecounter.nmicrotime: 2825851990 kern.timecounter.nnanotime: 4427452 kern.timecounter.nbintime: 2830279204 kern.timecounter.nmicrouptime: 1409332 kern.timecounter.nnanouptime: 1395210 kern.timecounter.nbinuptime: 293850150 kern.timecounter.stepwarnings: 0 kern.timecounter.smp_tsc: 0 kern.threads.thr_concurrency: 0 kern.threads.thr_scope: 0 kern.threads.virtual_cpu: 2 kern.threads.max_threads_hits: 0 kern.threads.max_groups_per_proc: 1500 kern.threads.max_threads_per_proc: 1500 kern.threads.debug: 0 kern.ccpu: 1948 kern.sched.runq_fuzz: 1 kern.sched.preemption: 1 kern.sched.kgfollowons: 0 kern.sched.pfollowons: 0 kern.sched.followon: 0 kern.sched.ipiwakeup.htt2: 0 kern.sched.ipiwakeup.onecpu: 0 kern.sched.ipiwakeup.useloop: 0 kern.sched.ipiwakeup.usemask: 1 kern.sched.ipiwakeup.delivered: 61457042 kern.sched.ipiwakeup.requested: 61445760 kern.sched.ipiwakeup.enabled: 1 kern.sched.quantum: 100000 kern.sched.name: 4BSD kern.devstat.version: 6 kern.devstat.generation: 147 kern.devstat.numdevs: 1 kern.kobj_methodcount: 89 kern.log_wakeups_per_second: 5 kern.msgbuf_clear: 0 kern.msgbuf: kern.always_console_output: 0 kern.log_console_output: 1 kern.smp.forward_roundrobin_enabled: 1 kern.smp.forward_signal_enabled: 1 kern.smp.cpus: 2 kern.smp.disabled: 0 kern.smp.active: 1 kern.smp.maxcpus: 16 kern.nselcoll: 346 kern.tty_nout: 12466877 kern.tty_nin: 14015 kern.drainwait: 300 kern.constty_wakeups_per_second: 5 kern.consmsgbuf_size: 8192 kern.consmute: 0 kern.console: consolectl,/ttyd0,consolectl, kern.minvnodes: 25000 kern.metadelay: 28 kern.dirdelay: 29 kern.filedelay: 30 kern.chroot_allow_open_directories: 1 kern.rpc.invalid: 0 kern.rpc.unexpected: 0 kern.rpc.timeouts: 0 kern.rpc.request: 0 kern.rpc.retries: 0 kern.elf32.fallback_brand: -1 kern.random.yarrow.gengateinterval: 10 kern.random.yarrow.bins: 10 kern.random.yarrow.fastthresh: 192 kern.random.yarrow.slowthresh: 256 kern.random.yarrow.slowoverthresh: 2 kern.random.sys.seeded: 1 kern.random.sys.harvest.ethernet: 1 kern.random.sys.harvest.point_to_point: 1 kern.random.sys.harvest.interrupt: 1 kern.random.sys.harvest.swi: 0 vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 37) Virtual Memory: (Total: 4464588K, Active 91256K) Real Memory: (Total: 80516K Active 39540K) Shared Virtual Memory: (Total: 33300K Active: 30736K) Shared Real Memory: (Total: 12308K Active: 11012K) Free Memory Pages: 115256K vm.loadavg: { 0.08 0.10 0.08 } vm.v_free_min: 6456 vm.v_free_target: 27223 vm.v_free_reserved: 1399 vm.v_inactive_target: 40834 vm.v_cache_min: 27223 vm.v_cache_max: 54446 vm.v_pageout_free_min: 34 vm.pageout_algorithm: 0 vm.swap_enabled: 1 vm.kmem_size_scale: 3 vm.kmem_size_max: 419430400 vm.kmem_size: 419430400 vm.nswapdev: 1 vm.dmmax: 32 vm.swap_async_max: 4 vm.zone_count: 71 vm.zone: ITEM SIZE LIMIT USED FREE REQUESTS FFS2 dinode: 256, 0, 31153, 2117, 2665117 FFS1 dinode: 128, 0, 0, 0, 0 FFS inode: 192, 0, 31153, 3227, 2665117 Mountpoints: 792, 0, 10, 5, 11 SWAPMETA: 288, 116519, 1, 12, 398 IPFW dynamic: 120, 0, 161, 273, 760623 divcb: 304, 49152, 0, 0, 0 rtentry: 264, 0, 7, 35, 37 unpcb: 200, 49153, 42, 262, 1842480 ripcb: 304, 49152, 0, 36, 152 sackhole: 32, 0, 13, 492, 217243049 tcpreass: 40, 2100, 0, 420, 247314 hostcache: 136, 15372, 2339, 209, 1587635 syncache: 128, 15370, 2, 143, 7427992 tcptw: 80, 8235, 42, 588, 2262469 tcpcb: 752, 49155, 193, 1357, 7474435 inpcb: 304, 49152, 235, 1373, 7474435 udpcb: 304, 49152, 7, 41, 1228897 ipq: 56, 1071, 0, 0, 40238 socket: 616, 49152, 242, 1672, 10545969 KNOTE: 120, 0, 101, 271, 6365204681 PIPE: 768, 0, 35, 110, 1054714 DIRHASH: 1024, 0, 203, 133, 4276 NFSNODE: 664, 0, 20, 34, 19966 NFSMOUNT: 584, 0, 2, 5, 2 L VFS Cache: 327, 0, 226, 314, 37258 S VFS Cache: 104, 0, 30642, 10938, 4538089 NAMEI: 1024, 0, 0, 260, 3822013249 VNODEPOLL: 152, 0, 0, 50, 1 VNODE: 496, 0, 31220, 18948, 2685452 ata_composit: 376, 0, 0, 0, 0 ata_request: 336, 0, 0, 22, 24 g_bio: 216, 0, 4, 266, 147855894 ACL UMA zone: 388, 0, 0, 0, 0 mbuf_jumbo_1: 16384, 0, 0, 0, 0 mbuf_jumbo_9: 9216, 0, 0, 0, 0 mbuf_jumbo_p: 4096, 0, 0, 0, 0 mbuf_cluster: 2048, 32768, 467, 111, 7519060757 mbuf: 256, 0, 2552, 433, 38882139786 mbuf_packet: 256, 0, 2473, 512, 8364630232 VMSPACE: 544, 0, 36, 76, 1421522 UPCALL: 88, 0, 3, 73, 3 KSEGRP: 136, 0, 161, 73, 161 THREAD: 608, 0, 165, 9, 153943 PROC: 856, 0, 83, 77, 1409292 Files: 120, 0, 296, 386, 3661475205 4096: 4096, 0, 240, 235, 2359833 2048: 2048, 0, 29, 161, 749612 1024: 1024, 0, 50, 222, 265791 512: 512, 0, 599, 255, 4069145 256: 256, 0, 609, 261, 38331006 128: 128, 0, 4928, 640, 13840622 64: 64, 0, 6319, 737, 23251477 32: 32, 0, 1279, 337, 3465149 16: 16, 0, 3660, 4404, 5890917059 mt_zone: 1024, 0, 170, 6, 170 DP fakepg: 120, 0, 0, 0, 0 PV ENTRY: 48, 2244600, 20590, 36218, 1346939535 MAP ENTRY: 112, 0, 1439, 1036, 107995821 KMAP ENTRY: 112, 90222, 292, 1490, 8014685 MAP: 352, 0, 7, 15, 7 VM OBJECT: 224, 0, 9271, 32957, 39999957 128 Bucket: 1048, 0, 248, 91, 4519 64 Bucket: 536, 0, 27, 43, 83 32 Bucket: 280, 0, 16, 54, 67 16 Bucket: 152, 0, 51, 49, 81 UMA Hash: 256, 0, 4, 11, 7 UMA RCntSlab: 128, 0, 289, 59, 998251 UMA Slabs: 128, 0, 881, 134, 880304 UMA Zones: 376, 0, 69, 1, 69 UMA Kegs: 240, 0, 69, 6, 69 vm.old_contigmalloc: 0 vm.swap_idle_threshold2: 10 vm.swap_idle_threshold1: 2 vm.exec_map_entries: 16 vm.stats.misc.zero_page_count: 1010 vm.stats.misc.cnt_prezero: 78684204 vm.stats.vm.v_kthreadpages: 0 vm.stats.vm.v_rforkpages: 0 vm.stats.vm.v_vforkpages: 11011007 vm.stats.vm.v_forkpages: 328234200 vm.stats.vm.v_kthreads: 53 vm.stats.vm.v_rforks: 0 vm.stats.vm.v_vforks: 114153 vm.stats.vm.v_forks: 1295086 vm.stats.vm.v_interrupt_free_min: 2 vm.stats.vm.v_pageout_free_min: 34 vm.stats.vm.v_cache_max: 54446 vm.stats.vm.v_cache_min: 27223 vm.stats.vm.v_cache_count: 27329 vm.stats.vm.v_inactive_count: 890787 vm.stats.vm.v_inactive_target: 40834 vm.stats.vm.v_active_count: 11742 vm.stats.vm.v_wire_count: 63202 vm.stats.vm.v_free_count: 1467 vm.stats.vm.v_free_min: 6456 vm.stats.vm.v_free_target: 27223 vm.stats.vm.v_free_reserved: 1399 vm.stats.vm.v_page_count: 1011794 vm.stats.vm.v_page_size: 4096 vm.stats.vm.v_tfree: 723316037 vm.stats.vm.v_pfree: 121750948 vm.stats.vm.v_dfree: 121 vm.stats.vm.v_pdpages: 546500114 vm.stats.vm.v_pdwakeups: 19084 vm.stats.vm.v_reactivated: 231346 vm.stats.vm.v_intrans: 3132 vm.stats.vm.v_vnodepgsout: 0 vm.stats.vm.v_vnodepgsin: 82081 vm.stats.vm.v_vnodeout: 0 vm.stats.vm.v_vnodein: 24268 vm.stats.vm.v_swappgsout: 377 vm.stats.vm.v_swappgsin: 339 vm.stats.vm.v_swapout: 374 vm.stats.vm.v_swapin: 335 vm.stats.vm.v_ozfod: 42609991 vm.stats.vm.v_zfod: 84134694 vm.stats.vm.v_cow_optim: 2529 vm.stats.vm.v_cow_faults: 71443486 vm.stats.vm.v_vm_faults: 208731149 vm.stats.sys.v_soft: 8869351 vm.stats.sys.v_intr: 908914123 vm.stats.sys.v_syscall: 1726552299 vm.stats.sys.v_trap: 2945088101 vm.stats.sys.v_swtch: 1463628763 vm.v_free_severe: 3927 vm.max_proc_mmap: 37449 vm.old_msync: 0 vm.msync_flush_flags: 3 vm.boot_pages: 48 vm.pageout_lock_miss: 0 vm.disable_swapspace_pageouts: 0 vm.defer_swapspace_pageouts: 0 vm.swap_idle_enabled: 0 vm.pageout_stats_interval: 5 vm.pageout_full_stats_interval: 20 vm.pageout_stats_max: 27223 vm.max_launder: 32 vm.idlezero_maxrun: 16 vm.idlezero_enable: 1 vm.kvm_free: 1226829824 vm.kvm_size: 2147479552 vfs.nfs.downdelayinitial: 12 vfs.nfs.downdelayinterval: 30 vfs.nfs.nfs3_jukebox_delay: 10 vfs.nfs.reconnects: 0 vfs.nfs.bufpackets: 4 vfs.nfs.realign_count: 0 vfs.nfs.realign_test: 241605 vfs.nfs.defect: 0 vfs.nfs.iodmax: 20 vfs.nfs.iodmin: 0 vfs.nfs.iodmaxidle: 120 vfs.nfs.diskless_rootpath: vfs.nfs.diskless_valid: 0 vfs.nfs.nfs_ip_paranoia: 1 vfs.nfs.nfs_directio_allow_mmap: 1 vfs.nfs.nfs_directio_enable: 0 vfs.nfs.clean_pages_on_close: 1 vfs.nfs.nfsv3_commit_on_close: 0 vfs.nfs.access_cache_timeout: 60 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 240482 vfs.ufs.dirhash_maxmem: 33554432 vfs.ufs.dirhash_minsize: 2560 vfs.devfs.rule_depth: 1 vfs.devfs.generation: 94 vfs.nfs4.nfsv3_commit_on_close: 0 vfs.nfs4.access_cache_timeout: 60 vfs.pfs.vncache.misses: 169 vfs.pfs.vncache.hits: 125 vfs.pfs.vncache.maxentries: 96 vfs.pfs.vncache.entries: 1 vfs.flushwithdeps: 0 vfs.getnewbufrestarts: 6850679 vfs.getnewbufcalls: 37407273 vfs.hifreebuffers: 1540 vfs.lofreebuffers: 770 vfs.numfreebuffers: 13754 vfs.dirtybufthresh: 3115 vfs.hidirtybuffers: 3462 vfs.lodirtybuffers: 1731 vfs.numdirtybuffers: 15 vfs.recursiveflushes: 0 vfs.altbufferflushes: 0 vfs.bdwriteskip: 0 vfs.dirtybufferflushes: 0 vfs.hirunningspace: 1048576 vfs.lorunningspace: 524288 vfs.bufdefragcnt: 3064630 vfs.buffreekvacnt: 4676556 vfs.bufreusecnt: 4680388 vfs.hibufspace: 224968704 vfs.lobufspace: 224903168 vfs.maxmallocbufspace: 11248435 vfs.bufmallocspace: 10240 vfs.maxbufspace: 225624064 vfs.bufspace: 217923584 vfs.runningbufspace: 0 vfs.vmiodirenable: 1 vfs.cache.numfullpathfound: 260787 vfs.cache.numfullpathfail4: 0 vfs.cache.numfullpathfail2: 0 vfs.cache.numfullpathfail1: 1 vfs.cache.numfullpathcalls: 260788 vfs.cache.nchstats: 26325721396 33299751 3068375 0 8638501 0 194947 364966 vfs.cache.numneghits: 33299751 vfs.cache.numnegzaps: 857827 vfs.cache.numposhits: 26325721396 vfs.cache.numposzaps: 2210548 vfs.cache.nummisszap: 2757267 vfs.cache.nummiss: 5881234 vfs.cache.numchecks: 31702575347 vfs.cache.dotdothits: 196200 vfs.cache.dothits: 161626 vfs.cache.numcalls: 26371085849 vfs.cache.numcache: 30868 vfs.cache.numneg: 1115 vfs.read_max: 8 vfs.write_behind: 1 vfs.lookup_shared: 0 vfs.usermount: 0 vfs.worklist_len: 5 vfs.timestamp_precision: 0 vfs.reassignbufcalls: 16981143 vfs.freevnodes: 24772 vfs.wantfreevnodes: 25000 vfs.numvnodes: 31220 vfs.nfsrv.nfs_privport: 0 vfs.nfsrv.commit_miss: 0 vfs.nfsrv.commit_blks: 0 vfs.nfsrv.async: 0 vfs.nfsrv.realign_count: 0 vfs.nfsrv.realign_test: 0 vfs.nfsrv.gatherdelay_v3: 0 vfs.nfsrv.gatherdelay: 10000 vfs.ffs.doreallocblks: 1 vfs.ffs.doasyncfree: 1 vfs.ffs.compute_summary_at_mount: 0 net.local.stream.recvspace: 8192 net.local.stream.sendspace: 8192 net.local.dgram.recvspace: 4096 net.local.dgram.maxdgram: 2048 net.local.recycled: 0 net.local.taskcount: 0 net.local.inflight: 0 net.inet.ip.portrange.randomtime: 45 net.inet.ip.portrange.randomcps: 10 net.inet.ip.portrange.randomized: 1 net.inet.ip.portrange.reservedlow: 0 net.inet.ip.portrange.reservedhigh: 1023 net.inet.ip.portrange.hilast: 65535 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.first: 49152 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.forwarding: 0 net.inet.ip.redirect: 0 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 3600 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 50 net.inet.ip.intr_queue_drops: 72078 net.inet.ip.accept_sourceroute: 0 net.inet.ip.keepfaith: 0 net.inet.ip.same_prefix_carp_only: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.dummynet.debug: 0 net.inet.ip.dummynet.tick_lost: 0 net.inet.ip.dummynet.tick_diff: 7104348 net.inet.ip.dummynet.tick_adjustment: 6728079 net.inet.ip.dummynet.tick_delta_sum: 168 net.inet.ip.dummynet.tick_delta: 8 net.inet.ip.dummynet.red_max_pkt_size: 1500 net.inet.ip.dummynet.red_avg_pkt_size: 512 net.inet.ip.dummynet.red_lookup_depth: 256 net.inet.ip.dummynet.max_chain_len: 16 net.inet.ip.dummynet.expire: 1 net.inet.ip.dummynet.search_steps: 0 net.inet.ip.dummynet.searches: 0 net.inet.ip.dummynet.extract_heap: 0 net.inet.ip.dummynet.ready_heap: 0 net.inet.ip.dummynet.curr_time: 4091820365 net.inet.ip.dummynet.hash_size: 64 net.inet.ip.fastforwarding: 0 net.inet.ip.fw.dyn_keepalive: 1 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.static_count: 19 net.inet.ip.fw.dyn_max: 4096 net.inet.ip.fw.dyn_count: 161 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.verbose: 1 net.inet.ip.fw.debug: 1 net.inet.ip.fw.one_pass: 1 net.inet.ip.fw.autoinc_step: 100 net.inet.ip.fw.enable: 1 net.inet.ip.maxfragpackets: 1024 net.inet.ip.maxfragsperpacket: 16 net.inet.ip.fragpackets: 0 net.inet.ip.check_interface: 0 net.inet.ip.random_id: 0 net.inet.ip.sendsourcequench: 0 net.inet.ip.process_options: 1 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 100 net.inet.icmp.bmcastecho: 0 net.inet.icmp.quotelen: 8 net.inet.icmp.reply_from_interface: 0 net.inet.icmp.reply_src: net.inet.icmp.icmplim_output: 1 net.inet.icmp.log_redirect: 0 net.inet.icmp.drop_redirect: 1 net.inet.icmp.maskfake: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 512 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 65536 net.inet.tcp.recvspace: 32768 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.hostcache.purge: 0 net.inet.tcp.hostcache.prune: 300 net.inet.tcp.hostcache.expire: 3600 net.inet.tcp.hostcache.count: 2339 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.reass.overflows: 0 net.inet.tcp.reass.maxqlen: 48 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.maxsegments: 2048 net.inet.tcp.insecure_rst: 0 net.inet.tcp.rfc3390: 1 net.inet.tcp.rfc3042: 1 net.inet.tcp.drop_synfin: 1 net.inet.tcp.delayed_ack: 1 net.inet.tcp.blackhole: 1 net.inet.tcp.log_in_vain: 0 net.inet.tcp.newreno: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.sack.globalholes: 14 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.enable: 1 net.inet.tcp.inflight.stab: 20 net.inet.tcp.inflight.max: 1073725440 net.inet.tcp.inflight.min: 6144 net.inet.tcp.inflight.rttthresh: 10 net.inet.tcp.inflight.debug: 0 net.inet.tcp.inflight.enable: 1 net.inet.tcp.nolocaltimewait: 0 net.inet.tcp.maxtcptw: 8191 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.pcbcount: 235 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.minmssoverload: 0 net.inet.tcp.minmss: 216 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.count: 2 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncookies: 1 net.inet.tcp.always_keepalive: 1 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.rexmit_min: 3 net.inet.tcp.msl: 30000 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 41600 net.inet.udp.strict_mcast_mship: 0 net.inet.udp.blackhole: 1 net.inet.udp.log_in_vain: 0 net.inet.carp.allow: 1 net.inet.carp.preempt: 0 net.inet.carp.log: 1 net.inet.carp.arpbalance: 0 net.inet.carp.suppress_preempt: 0 net.inet.raw.recvspace: 9216 net.inet.raw.maxdgram: 9216 net.inet.ipf.fr_minttl: 4 net.inet.ipf.fr_chksrc: 0 net.inet.ipf.fr_defaultauthage: 600 net.inet.ipf.fr_authused: 0 net.inet.ipf.fr_authsize: 32 net.inet.ipf.ipf_hostmap_sz: 2047 net.inet.ipf.ipf_rdrrules_sz: 127 net.inet.ipf.ipf_natrules_sz: 127 net.inet.ipf.ipf_nattable_sz: 2047 net.inet.ipf.fr_statemax: 4013 net.inet.ipf.fr_statesize: 5737 net.inet.ipf.fr_running: 1 net.inet.ipf.fr_ipfrttl: 120 net.inet.ipf.fr_defnatage: 1200 net.inet.ipf.fr_icmptimeout: 120 net.inet.ipf.fr_udpacktimeout: 24 net.inet.ipf.fr_udptimeout: 240 net.inet.ipf.fr_tcpclosed: 120 net.inet.ipf.fr_tcptimeout: 480 net.inet.ipf.fr_tcplastack: 480 net.inet.ipf.fr_tcpclosewait: 480 net.inet.ipf.fr_tcphalfclosed: 14400 net.inet.ipf.fr_tcpidletimeout: 864000 net.inet.ipf.fr_active: 0 net.inet.ipf.fr_pass: 134217730 net.inet.ipf.fr_flags: 0 net.inet.accf.unloadable: 0 net.inet.accf.http.parsehttpversion: 1 net.link.generic.system.ifcount: 3 net.link.ether.inet.log_arp_permanent_modify: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.proxyall: 0 net.link.ether.inet.useloopback: 1 net.link.ether.inet.maxtries: 5 net.link.ether.inet.max_age: 1200 net.link.ether.inet.prune_intvl: 300 net.link.ether.ipfw: 0 net.link.log_link_state_change: 1 net.link.tun.devfs_cloning: 1 net.bpf.maxinsns: 512 net.bpf.maxbufsize: 524288 net.bpf.bufsize: 4096 net.isr.swi_count: 1577955123 net.isr.drop: 0 net.isr.queued: 8308 net.isr.deferred: -261229993 net.isr.directed: 0 net.isr.count: -261229992 net.isr.direct: 0 net.route.netisr_maxqlen: 256 debug.acpi.semaphore_debug: 0 debug.acpi.suspend_bounce: 0 debug.acpi.do_powerstate: 1 debug.acpi.acpi_ca_version: 0x20041119 debug.mddebug: 0 debug.elf64_legacy_coredump: 0 debug.elf64_trace: 0 debug.bootverbose: 0 debug.boothowto: 0 debug.cpufreq.verbose: 0 debug.cpufreq.lowest: 0 debug.sizeof.cdev_priv: 336 debug.sizeof.cdev: 288 debug.sizeof.g_bioq: 96 debug.sizeof.g_consumer: 96 debug.sizeof.g_provider: 136 debug.sizeof.g_geom: 136 debug.sizeof.g_class: 136 debug.sizeof.kinfo_proc: 1088 debug.sizeof.buf: 584 debug.sizeof.bio: 216 debug.sizeof.proc: 856 debug.sizeof.vnode: 496 debug.sizeof.devstat: 288 debug.to_avg_mpcalls: 1209 debug.to_avg_mtxcalls: 0 debug.to_avg_gcalls: 0 debug.to_avg_depth: 1251 debug.kdb.stop_cpus: 1 debug.kdb.enter: 0 debug.kdb.current: debug.kdb.available: debug.rman_debug: 0 debug.ttydebug: 0 debug.disablefullpath: 0 debug.disablecwd: 0 debug.hashstat.nchash: 131072 27468 4 2095 debug.ncsize: 72 debug.vnsize: 496 debug.vfscache: 1 debug.numcachehv: 4757 debug.numcache: 30868 debug.numneg: 1115 debug.ncnegfactor: 16 debug.nchash: 131071 debug.vnlru_nowhere: 0 debug.rush_requests: 0 debug.mpsafevfs: 1 debug.if_tun_debug: 0 debug.mpsafenet: 1 debug.collectsnapstats: 0 debug.snapdebug: 0 debug.dopersistence: 0 debug.dir_entry: 19483 debug.direct_blk_ptrs: 22023 debug.inode_bitmap: 13973 debug.indir_blk_ptrs: 75044 debug.sync_limit_hit: 0 debug.ino_limit_hit: 0 debug.blk_limit_hit: 0 debug.ino_limit_push: 0 debug.blk_limit_push: 0 debug.worklist_push: 0 debug.maxindirdeps: 50 debug.tickdelay: 2 debug.max_softdeps: 400000 debug.dobkgrdwrite: 1 debug.bigcgs: 0 debug.dircheck: 0 debug.nosleepwithlocks: 0 debug.mpsafevm: 1 debug.minidump: 0 debug.fdc.settle: 125 debug.fdc.spec2: 16 debug.fdc.spec1: 175 debug.fdc.retries: 10 debug.fdc.debugflags: 0 debug.fdc.fifo: 8 debug.elf32_legacy_coredump: 0 debug.elf32_trace: 0 hw.machine: amd64 hw.model: Intel(R) Xeon(TM) CPU 3.20GHz hw.ncpu: 2 hw.byteorder: 1234 hw.physmem: 4286275584 hw.usermem: 4027437056 hw.pagesize: 4096 hw.floatingpoint: 1 hw.machine_arch: amd64 hw.realmem: 4831838208 hw.amr.force_sg32: 0 hw.ata.wc: 1 hw.ata.atapi_dma: 1 hw.ata.ata_dma: 1 hw.bce.msi_enable: 1 hw.bge.allow_asf: 0 hw.bge.fake_autoneg: 0 hw.mfi.event_class: 0 hw.mfi.event_locale: 65535 hw.pci.honor_msi_blacklist: 1 hw.pci.enable_msix: 0 hw.pci.enable_msi: 0 hw.pci.do_power_resume: 1 hw.pci.do_power_nodriver: 0 hw.pci.enable_io_modes: 1 hw.pci.host_mem_start: 2147483648 hw.intr_storm_threshold: 1000 hw.availpages: 1046454 hw.bus.devctl_disable: 0 hw.busdma.total_bpages: 3905 hw.busdma.zone0.total_bpages: 3872 hw.busdma.zone0.free_bpages: 3872 hw.busdma.zone0.reserved_bpages: 0 hw.busdma.zone0.active_bpages: 0 hw.busdma.zone0.total_bounced: 971252 hw.busdma.zone0.total_deferred: 0 hw.busdma.zone0.lowaddr: 0xffffffff hw.busdma.zone0.alignment: 1 hw.busdma.zone0.boundary: 0 hw.busdma.zone1.total_bpages: 1 hw.busdma.zone1.free_bpages: 1 hw.busdma.zone1.reserved_bpages: 0 hw.busdma.zone1.active_bpages: 0 hw.busdma.zone1.total_bounced: 0 hw.busdma.zone1.total_deferred: 0 hw.busdma.zone1.lowaddr: 0xffffffff hw.busdma.zone1.alignment: 4096 hw.busdma.zone1.boundary: 0 hw.busdma.zone2.total_bpages: 32 hw.busdma.zone2.free_bpages: 32 hw.busdma.zone2.reserved_bpages: 0 hw.busdma.zone2.active_bpages: 0 hw.busdma.zone2.total_bounced: 0 hw.busdma.zone2.total_deferred: 0 hw.busdma.zone2.lowaddr: 0xffffffff hw.busdma.zone2.alignment: 2 hw.busdma.zone2.boundary: 65536 hw.clockrate: 3192 hw.instruction_sse: 1 hw.apic.enable_extint: 0 hw.kbd.keymap_restrict_change: 0 hw.syscons.kbd_debug: 1 hw.syscons.kbd_reboot: 1 hw.syscons.bell: 1 hw.syscons.saver.keybonly: 1 hw.syscons.sc_no_suspend_vtswitch: 0 hw.acpi.supported_sleep_state: S4 S5 hw.acpi.power_button_state: S5 hw.acpi.sleep_button_state: S4 hw.acpi.lid_switch_state: NONE hw.acpi.standby_state: S1 hw.acpi.suspend_state: S3 hw.acpi.sleep_delay: 1 hw.acpi.s4bios: 0 hw.acpi.verbose: 0 hw.acpi.disable_on_reboot: 0 hw.acpi.handle_reboot: 0 hw.acpi.cpu.cx_lowest: C1 machdep.adjkerntz: -14400 machdep.disable_rtc_set: 0 machdep.wall_cmos_clock: 1 machdep.acpi_timer_freq: 3579545 machdep.acpi_root: 1037744 machdep.disable_mtrrs: 0 machdep.cpu_idle_hlt: 1 machdep.hlt_cpus: 0 machdep.panic_on_nmi: 1 machdep.tsc_freq: 3192012016 machdep.i8254_freq: 1193182 machdep.conspeed: 9600 machdep.gdbspeed: 9600 machdep.conrclk: 1843200 machdep.enable_panic_key: 0 user.cs_path: /usr/bin:/bin:/usr/sbin:/sbin: user.bc_base_max: 99 user.bc_dim_max: 2048 user.bc_scale_max: 99 user.bc_string_max: 1000 user.coll_weights_max: 0 user.expr_nest_max: 32 user.line_max: 2048 user.re_dup_max: 255 user.posix2_version: 199212 user.posix2_c_bind: 0 user.posix2_c_dev: 0 user.posix2_char_term: 0 user.posix2_fort_dev: 0 user.posix2_fort_run: 0 user.posix2_localedef: 0 user.posix2_sw_dev: 0 user.posix2_upe: 0 user.stream_max: 20 user.tzname_max: 255 p1003_1b.asynchronous_io: 0 p1003_1b.mapped_files: 1 p1003_1b.memlock: 0 p1003_1b.memlock_range: 0 p1003_1b.memory_protection: 0 p1003_1b.message_passing: 0 p1003_1b.prioritized_io: 0 p1003_1b.priority_scheduling: 1 p1003_1b.realtime_signals: 0 p1003_1b.semaphores: 0 p1003_1b.fsync: 0 p1003_1b.shared_memory_objects: 1 p1003_1b.synchronized_io: 0 p1003_1b.timers: 0 p1003_1b.aio_listio_max: -1 p1003_1b.aio_max: -1 p1003_1b.aio_prio_delta_max: -1 p1003_1b.delaytimer_max: 0 p1003_1b.mq_open_max: 0 p1003_1b.pagesize: 4096 p1003_1b.rtsig_max: 0 p1003_1b.sem_nsems_max: 0 p1003_1b.sem_value_max: 0 p1003_1b.sigqueue_max: 0 p1003_1b.timer_max: 0 security.jail.jailed: 0 security.jail.chflags_allowed: 0 security.jail.allow_raw_sockets: 1 security.jail.enforce_statfs: 2 security.jail.sysvipc_allowed: 0 security.jail.socket_unixiproute_only: 1 security.jail.set_hostname_allowed: 1 security.bsd.unprivileged_proc_debug: 1 security.bsd.conservative_signals: 1 security.bsd.see_other_gids: 1 security.bsd.see_other_uids: 1 security.bsd.suser_enabled: 1 security.bsd.unprivileged_read_msgbuf: 1 security.bsd.hardlink_check_gid: 0 security.bsd.hardlink_check_uid: 0 security.bsd.unprivileged_get_quota: 0 compat.ia32.maxvmem: 0 compat.ia32.maxssiz: 67108864 compat.ia32.maxdsiz: 536870912 compat.linux32.maxvmem: 0 compat.linux32.maxssiz: 67108864 compat.linux32.maxdsiz: 536870912 compat.linux.oss_version: 198144 compat.linux.osrelease: 2.4.2 compat.linux.osname: Linux dev.nexus.0.%driver: nexus dev.nexus.0.%parent: root0 dev.acpi.0.%desc: DELL PE BKC dev.acpi.0.%driver: acpi dev.acpi.0.%parent: nexus0 dev.acpi_hpet.0.%desc: High Precision Event Timer dev.acpi_hpet.0.%driver: acpi_hpet dev.acpi_hpet.0.%location: unknown dev.acpi_hpet.0.%pnpinfo: unknown dev.acpi_hpet.0.%parent: acpi0 dev.acpi_sysresource.0.%desc: System Resource dev.acpi_sysresource.0.%driver: acpi_sysresource dev.acpi_sysresource.0.%location: handle=\_SB_.PCI0.ISA_.MBIO dev.acpi_sysresource.0.%pnpinfo: _HID=PNP0C01 _UID=0 dev.acpi_sysresource.0.%parent: acpi0 dev.acpi_sysresource.1.%desc: System Resource dev.acpi_sysresource.1.%driver: acpi_sysresource dev.acpi_sysresource.1.%location: handle=\_SB_.PCI0.ISA_.MBI1 dev.acpi_sysresource.1.%pnpinfo: _HID=PNP0C01 _UID=1 dev.acpi_sysresource.1.%parent: acpi0 dev.acpi_sysresource.2.%desc: System Resource dev.acpi_sysresource.2.%driver: acpi_sysresource dev.acpi_sysresource.2.%location: handle=\_SB_.PCI0.PEHB dev.acpi_sysresource.2.%pnpinfo: _HID=PNP0C02 _UID=0 dev.acpi_sysresource.2.%parent: acpi0 dev.acpi_timer.0.%desc: 24-bit timer at 3.579545MHz dev.acpi_timer.0.%driver: acpi_timer dev.acpi_timer.0.%location: unknown dev.acpi_timer.0.%pnpinfo: unknown dev.acpi_timer.0.%parent: acpi0 dev.pci_link.0.%desc: ACPI PCI Link LNKA dev.pci_link.0.%driver: pci_link dev.pci_link.0.%location: handle=\_SB_.LNKA dev.pci_link.0.%pnpinfo: _HID=PNP0C0F _UID=1 dev.pci_link.0.%parent: acpi0 dev.pci_link.1.%desc: ACPI PCI Link LNKB dev.pci_link.1.%driver: pci_link dev.pci_link.1.%location: handle=\_SB_.LNKB dev.pci_link.1.%pnpinfo: _HID=PNP0C0F _UID=2 dev.pci_link.1.%parent: acpi0 dev.pci_link.2.%desc: ACPI PCI Link LNKC dev.pci_link.2.%driver: pci_link dev.pci_link.2.%location: handle=\_SB_.LNKC dev.pci_link.2.%pnpinfo: _HID=PNP0C0F _UID=3 dev.pci_link.2.%parent: acpi0 dev.pci_link.3.%desc: ACPI PCI Link LNKD dev.pci_link.3.%driver: pci_link dev.pci_link.3.%location: handle=\_SB_.LNKD dev.pci_link.3.%pnpinfo: _HID=PNP0C0F _UID=4 dev.pci_link.3.%parent: acpi0 dev.pci_link.4.%desc: ACPI PCI Link LNKE dev.pci_link.4.%driver: pci_link dev.pci_link.4.%location: handle=\_SB_.LNKE dev.pci_link.4.%pnpinfo: _HID=PNP0C0F _UID=5 dev.pci_link.4.%parent: acpi0 dev.pci_link.5.%desc: ACPI PCI Link LNKF dev.pci_link.5.%driver: pci_link dev.pci_link.5.%location: handle=\_SB_.LNKF dev.pci_link.5.%pnpinfo: _HID=PNP0C0F _UID=6 dev.pci_link.5.%parent: acpi0 dev.pci_link.6.%desc: ACPI PCI Link LNKG dev.pci_link.6.%driver: pci_link dev.pci_link.6.%location: handle=\_SB_.LNKG dev.pci_link.6.%pnpinfo: _HID=PNP0C0F _UID=7 dev.pci_link.6.%parent: acpi0 dev.pci_link.7.%desc: ACPI PCI Link LNKH dev.pci_link.7.%driver: pci_link dev.pci_link.7.%location: handle=\_SB_.LNKH dev.pci_link.7.%pnpinfo: _HID=PNP0C0F _UID=8 dev.pci_link.7.%parent: acpi0 dev.cpu.0.%desc: ACPI CPU dev.cpu.0.%driver: cpu dev.cpu.0.%location: handle=\_PR_.CPU0 dev.cpu.0.%pnpinfo: _HID=none _UID=0 dev.cpu.0.%parent: acpi0 dev.cpu.0.cx_supported: C1/0 dev.cpu.0.cx_lowest: C1 dev.cpu.0.cx_usage: 100.00% dev.cpu.1.%desc: ACPI CPU dev.cpu.1.%driver: cpu dev.cpu.1.%location: handle=\_PR_.CPU1 dev.cpu.1.%pnpinfo: _HID=none _UID=0 dev.cpu.1.%parent: acpi0 dev.cpu.1.cx_supported: C1/0 dev.cpu.1.cx_lowest: C1 dev.cpu.1.cx_usage: 100.00% dev.pcib.0.%desc: ACPI Host-PCI bridge dev.pcib.0.%driver: pcib dev.pcib.0.%location: handle=\_SB_.PCI0 dev.pcib.0.%pnpinfo: _HID=PNP0A03 _UID=1 dev.pcib.0.%parent: acpi0 dev.pcib.0.wake: 0 dev.pcib.1.%desc: ACPI PCI-PCI bridge dev.pcib.1.%driver: pcib dev.pcib.1.%location: slot=2 function=0 handle=\_SB_.PCI0.PALO dev.pcib.1.%pnpinfo: vendor=0x8086 device=0x3595 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.1.%parent: pci0 dev.pcib.1.wake: 0 dev.pcib.2.%desc: ACPI PCI-PCI bridge dev.pcib.2.%driver: pcib dev.pcib.2.%location: slot=0 function=0 handle=\_SB_.PCI0.PALO.DOBA dev.pcib.2.%pnpinfo: vendor=0x8086 device=0x0330 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.2.%parent: pci1 dev.pcib.3.%desc: ACPI PCI-PCI bridge dev.pcib.3.%driver: pcib dev.pcib.3.%location: slot=0 function=2 handle=\_SB_.PCI0.PALO.DOBB dev.pcib.3.%pnpinfo: vendor=0x8086 device=0x0332 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.3.%parent: pci1 dev.pcib.4.%desc: ACPI PCI-PCI bridge dev.pcib.4.%driver: pcib dev.pcib.4.%location: slot=4 function=0 handle=\_SB_.PCI0.PBLO dev.pcib.4.%pnpinfo: vendor=0x8086 device=0x3597 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.4.%parent: pci0 dev.pcib.4.wake: 0 dev.pcib.5.%desc: ACPI PCI-PCI bridge dev.pcib.5.%driver: pcib dev.pcib.5.%location: slot=5 function=0 handle=\_SB_.PCI0.PBHI dev.pcib.5.%pnpinfo: vendor=0x8086 device=0x3598 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.5.%parent: pci0 dev.pcib.5.wake: 0 dev.pcib.6.%desc: ACPI PCI-PCI bridge dev.pcib.6.%driver: pcib dev.pcib.6.%location: slot=0 function=0 handle=\_SB_.PCI0.PBHI.PXB1 dev.pcib.6.%pnpinfo: vendor=0x8086 device=0x0329 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.6.%parent: pci5 dev.pcib.7.%desc: ACPI PCI-PCI bridge dev.pcib.7.%driver: pcib dev.pcib.7.%location: slot=0 function=2 handle=\_SB_.PCI0.PBHI.PXB2 dev.pcib.7.%pnpinfo: vendor=0x8086 device=0x032a subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.7.%parent: pci5 dev.pcib.8.%desc: ACPI PCI-PCI bridge dev.pcib.8.%driver: pcib dev.pcib.8.%location: slot=6 function=0 handle=\_SB_.PCI0.VPR1 dev.pcib.8.%pnpinfo: vendor=0x8086 device=0x3599 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.8.%parent: pci0 dev.pcib.8.wake: 0 dev.pcib.9.%desc: ACPI PCI-PCI bridge dev.pcib.9.%driver: pcib dev.pcib.9.%location: slot=0 function=0 handle=\_SB_.PCI0.VPR1.PXC1 dev.pcib.9.%pnpinfo: vendor=0x8086 device=0x0329 subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.9.%parent: pci8 dev.pcib.10.%desc: ACPI PCI-PCI bridge dev.pcib.10.%driver: pcib dev.pcib.10.%location: slot=0 function=2 handle=\_SB_.PCI0.VPR1.PXC2 dev.pcib.10.%pnpinfo: vendor=0x8086 device=0x032a subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.10.%parent: pci8 dev.pcib.11.%desc: ACPI PCI-PCI bridge dev.pcib.11.%driver: pcib dev.pcib.11.%location: slot=30 function=0 handle=\_SB_.PCI0.PICH dev.pcib.11.%pnpinfo: vendor=0x8086 device=0x244e subvendor=0x0000 subdevice=0x0000 class=0x060400 dev.pcib.11.%parent: pci0 dev.pcib.11.wake: 0 dev.pci.0.%desc: ACPI PCI bus dev.pci.0.%driver: pci dev.pci.0.%parent: pcib0 dev.pci.0.wake: 0 dev.pci.1.%desc: ACPI PCI bus dev.pci.1.%driver: pci dev.pci.1.%parent: pcib1 dev.pci.1.wake: 0 dev.pci.2.%desc: ACPI PCI bus dev.pci.2.%driver: pci dev.pci.2.%parent: pcib2 dev.pci.3.%desc: ACPI PCI bus dev.pci.3.%driver: pci dev.pci.3.%parent: pcib3 dev.pci.4.%desc: ACPI PCI bus dev.pci.4.%driver: pci dev.pci.4.%parent: pcib4 dev.pci.4.wake: 0 dev.pci.5.%desc: ACPI PCI bus dev.pci.5.%driver: pci dev.pci.5.%parent: pcib5 dev.pci.5.wake: 0 dev.pci.6.%desc: ACPI PCI bus dev.pci.6.%driver: pci dev.pci.6.%parent: pcib6 dev.pci.7.%desc: ACPI PCI bus dev.pci.7.%driver: pci dev.pci.7.%parent: pcib7 dev.pci.8.%desc: ACPI PCI bus dev.pci.8.%driver: pci dev.pci.8.%parent: pcib8 dev.pci.8.wake: 0 dev.pci.9.%desc: ACPI PCI bus dev.pci.9.%driver: pci dev.pci.9.%parent: pcib9 dev.pci.10.%desc: ACPI PCI bus dev.pci.10.%driver: pci dev.pci.10.%parent: pcib10 dev.pci.11.%desc: ACPI PCI bus dev.pci.11.%driver: pci dev.pci.11.%parent: pcib11 dev.pci.11.wake: 0 dev.hostb.0.%desc: Host to PCI bridge dev.hostb.0.%driver: hostb dev.hostb.0.%location: slot=0 function=0 dev.hostb.0.%pnpinfo: vendor=0x8086 device=0x3590 subvendor=0x1028 subdevice=0x016d class=0x060000 dev.hostb.0.%parent: pci0 dev.amr.0.%desc: LSILogic MegaRAID 1.53 dev.amr.0.%driver: amr dev.amr.0.%location: slot=14 function=0 dev.amr.0.%pnpinfo: vendor=0x1028 device=0x0013 subvendor=0x1028 subdevice=0x016d class=0x010400 dev.amr.0.%parent: pci2 dev.amr.0.allow_volume_configure: 0 dev.em.0.%desc: Intel(R) PRO/1000 Network Connection Version - 6.2.9 dev.em.0.%driver: em dev.em.0.%location: slot=7 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1076 subvendor=0x1028 subdevice=0x016d class=0x020000 dev.em.0.%parent: pci6 dev.em.0.debug_info: -1 dev.em.0.stats: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection Version - 6.2.9 dev.em.1.%driver: em dev.em.1.%location: slot=8 function=0 dev.em.1.%pnpinfo: vendor=0x8086 device=0x1076 subvendor=0x1028 subdevice=0x016d class=0x020000 dev.em.1.%parent: pci7 dev.em.1.debug_info: -1 dev.em.1.stats: -1 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.uhci.0.%desc: Intel 82801EB (ICH5) USB controller USB-A dev.uhci.0.%driver: uhci dev.uhci.0.%location: slot=29 function=0 dev.uhci.0.%pnpinfo: vendor=0x8086 device=0x24d2 subvendor=0x1028 subdevice=0x016d class=0x0c0300 dev.uhci.0.%parent: pci0 dev.uhci.1.%desc: Intel 82801EB (ICH5) USB controller USB-B dev.uhci.1.%driver: uhci dev.uhci.1.%location: slot=29 function=1 dev.uhci.1.%pnpinfo: vendor=0x8086 device=0x24d4 subvendor=0x1028 subdevice=0x016d class=0x0c0300 dev.uhci.1.%parent: pci0 dev.uhci.2.%desc: Intel 82801EB (ICH5) USB controller USB-C dev.uhci.2.%driver: uhci dev.uhci.2.%location: slot=29 function=2 dev.uhci.2.%pnpinfo: vendor=0x8086 device=0x24d7 subvendor=0x1028 subdevice=0x016d class=0x0c0300 dev.uhci.2.%parent: pci0 dev.usb.0.%desc: Intel 82801EB (ICH5) USB controller USB-A dev.usb.0.%driver: usb dev.usb.0.%parent: uhci0 dev.usb.1.%desc: Intel 82801EB (ICH5) USB controller USB-B dev.usb.1.%driver: usb dev.usb.1.%parent: uhci1 dev.usb.2.%desc: Intel 82801EB (ICH5) USB controller USB-C dev.usb.2.%driver: usb dev.usb.2.%parent: uhci2 dev.usb.3.%desc: Intel 82801EB/R (ICH5) USB 2.0 controller dev.usb.3.%driver: usb dev.usb.3.%parent: ehci0 dev.uhub.0.%desc: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 dev.uhub.0.%driver: uhub dev.uhub.0.%parent: usb0 dev.uhub.1.%desc: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 dev.uhub.1.%driver: uhub dev.uhub.1.%parent: usb1 dev.uhub.2.%desc: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 dev.uhub.2.%driver: uhub dev.uhub.2.%parent: usb2 dev.uhub.3.%desc: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 dev.uhub.3.%driver: uhub dev.uhub.3.%parent: usb3 dev.uhub.4.%desc: vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2 dev.uhub.4.%driver: uhub dev.uhub.4.%location: port=2 dev.uhub.4.%pnpinfo: vendor=0x413c product=0xa001 devclass=0x09 devsubclass=0x00 release=0x0000 sernum="" dev.uhub.4.%parent: uhub3 dev.ehci.0.%desc: Intel 82801EB/R (ICH5) USB 2.0 controller dev.ehci.0.%driver: ehci dev.ehci.0.%location: slot=29 function=7 dev.ehci.0.%pnpinfo: vendor=0x8086 device=0x24dd subvendor=0x1028 subdevice=0x016d class=0x0c0320 dev.ehci.0.%parent: pci0 dev.isab.0.%desc: PCI-ISA bridge dev.isab.0.%driver: isab dev.isab.0.%location: slot=31 function=0 handle=\_SB_.PCI0.ISA_ dev.isab.0.%pnpinfo: vendor=0x8086 device=0x24d0 subvendor=0x0000 subdevice=0x0000 class=0x060100 dev.isab.0.%parent: pci0 dev.isa.0.%desc: ISA bus dev.isa.0.%driver: isa dev.isa.0.%parent: isab0 dev.atapci.0.%desc: Intel ICH5 UDMA100 controller dev.atapci.0.%driver: atapci dev.atapci.0.%location: slot=31 function=1 dev.atapci.0.%pnpinfo: vendor=0x8086 device=0x24db subvendor=0x1028 subdevice=0x016d class=0x01018a dev.atapci.0.%parent: pci0 dev.ata.0.%desc: ATA channel 0 dev.ata.0.%driver: ata dev.ata.0.%parent: atapci0 dev.ata.1.%desc: ATA channel 1 dev.ata.1.%driver: ata dev.ata.1.%parent: atapci0 dev.atdma.0.%desc: AT DMA controller dev.atdma.0.%driver: atdma dev.atdma.0.%location: handle=\_SB_.PCI0.ISA_.DMA_ dev.atdma.0.%pnpinfo: _HID=PNP0200 _UID=0 dev.atdma.0.%parent: acpi0 dev.fpupnp.0.%desc: Legacy ISA coprocessor support dev.fpupnp.0.%driver: fpupnp dev.fpupnp.0.%location: handle=\_SB_.PCI0.ISA_.FPU_ dev.fpupnp.0.%pnpinfo: _HID=PNP0C04 _UID=0 dev.fpupnp.0.%parent: acpi0 dev.attimer.0.%desc: AT realtime clock dev.attimer.0.%driver: attimer dev.attimer.0.%location: handle=\_SB_.PCI0.ISA_.RTC_ dev.attimer.0.%pnpinfo: _HID=PNP0B00 _UID=0 dev.attimer.0.%parent: acpi0 dev.attimer.1.%desc: AT timer dev.attimer.1.%driver: attimer dev.attimer.1.%location: handle=\_SB_.PCI0.ISA_.TMR_ dev.attimer.1.%pnpinfo: _HID=PNP0100 _UID=0 dev.attimer.1.%parent: acpi0 dev.fdc.0.%desc: Enhanced floppy controller dev.fdc.0.%driver: fdc dev.fdc.0.%location: handle=\_SB_.PCI0.ISA_.FDC_ dev.fdc.0.%pnpinfo: _HID=PNP0700 _UID=0 dev.fdc.0.%parent: acpi0 dev.fd.0.%desc: 1440-KB 3.5" drive dev.fd.0.%driver: fd dev.fd.0.%parent: fdc0 dev.sio.0.%desc: 16550A-compatible COM port dev.sio.0.%driver: sio dev.sio.0.%location: handle=\_SB_.PCI0.ISA_.COMA dev.sio.0.%pnpinfo: _HID=PNP0501 _UID=1 dev.sio.0.%parent: acpi0 dev.orm.0.%desc: ISA Option ROMs dev.orm.0.%driver: orm dev.orm.0.%parent: isa0 dev.atkbdc.0.%desc: Keyboard controller (i8042) dev.atkbdc.0.%driver: atkbdc dev.atkbdc.0.%parent: isa0 dev.atkbd.0.%desc: AT Keyboard dev.atkbd.0.%driver: atkbd dev.atkbd.0.%parent: atkbdc0 dev.sc.0.%desc: System console dev.sc.0.%driver: sc dev.sc.0.%parent: isa0 dev.vga.0.%desc: Generic ISA VGA dev.vga.0.%driver: vga dev.vga.0.%parent: isa0 dev.acd.0.%desc: HL-DT-STCD-RW/DVD-ROM GCC-4244N/B101 dev.acd.0.%driver: acd dev.acd.0.%parent: ata0 dev.amrd.0.%desc: LSILogic MegaRAID logical drive dev.amrd.0.%driver: amrd dev.amrd.0.%parent: amr0 --------------040306070507060607050303-- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 09:06:15 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6683116A468; Tue, 16 Oct 2007 09:06:15 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id 872B813C48A; Tue, 16 Oct 2007 09:06:14 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.13) with ESMTPA id 194327216; Tue, 16 Oct 2007 13:06:20 +0400 Message-ID: <47147ED7.7060806@chistydom.ru> Date: Tue, 16 Oct 2007 13:05:27 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 Newsgroups: gmane.os.freebsd.stable,gmane.os.freebsd.devel.hackers To: Krassimir Slavchev References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147B95.9040400@bulinfo.net> In-Reply-To: <47147B95.9040400@bulinfo.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 16 Oct 2007 11:22:27 +0000 Cc: freebsd-hackers@freebsd.org, Kris Kennaway , freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 09:06:15 -0000 Hi. Krassimir Slavchev wrote: > You run apache with mod_perl or php too. How many clients handle this > apache server? Also in this light load you have locked files! Check > script execution times (/server-status may be useful). > When you have hight load check swap usage and haw many processes are in > lockf state. Apache is not much used here, it is just for kind of content management. It is not exposed to external users. With best regards, Alexey Popov From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 09:15:19 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7CAA516A420; Tue, 16 Oct 2007 09:15:19 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from mx.bulinfo.net (mx.bulinfo.net [193.194.156.1]) by mx1.freebsd.org (Postfix) with ESMTP id 23E1D13C491; Tue, 16 Oct 2007 09:15:17 +0000 (UTC) (envelope-from krassi@bulinfo.net) Received: from localhost (localhost [127.0.0.1]) by mx.bulinfo.net (Postfix) with ESMTP id 7748316BC2; Tue, 16 Oct 2007 11:51:36 +0300 (EEST) Received: from mx.bulinfo.net ([127.0.0.1]) by localhost (mx.bulinfo.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 41083-03; Tue, 16 Oct 2007 11:51:34 +0300 (EEST) Received: from [192.168.2.188] (pythia.bulinfo.net [212.72.195.5]) by mx.bulinfo.net (Postfix) with ESMTP id 6239E16BBB; Tue, 16 Oct 2007 11:51:34 +0300 (EEST) Message-ID: <47147B95.9040400@bulinfo.net> Date: Tue, 16 Oct 2007 11:51:33 +0300 From: Krassimir Slavchev User-Agent: Thunderbird 2.0.0.6 (X11/20070927) MIME-Version: 1.0 To: Alexey Popov References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> In-Reply-To: <47146FB4.6040306@chistydom.ru> X-Enigmail-Version: 0.95.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at mx.bulinfo.net X-Mailman-Approved-At: Tue, 16 Oct 2007 11:22:27 +0000 Cc: freebsd-hackers@freebsd.org, Kris Kennaway , freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 09:15:19 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexey Popov wrote: > Hi. > > Kris Kennaway wrote: > >>> After some time of running under high load disk performance become >>> expremely poor. At that periods 'systat -vm 1' shows something like >>> this: >> What does "high load" mean? You need to explain the system workload >> more. > This web service is similiar to YouTube. This server is video store. I > have around 200G of *.flv (flash video) files on the server. > > I run lighttpd as a web server. Disk load is usually around 50%, network > output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. > > As you can see it is a trivial service - sending files to network via HTTP. > >> >>> Disks amrd0 >>> KB/t 85.39 >>> tps 5 >>> MB/s 0.38 >>> % busy 99 >> >>> Apart of all, I tried to make mutex profiling and here's the results >>> (sorted by the total number of acquisitions): >>> >>> Bad case: >>> >>> 102 223514 273977 0 14689 1651568 /usr/src/sys/vm/uma_core.c:2349 (512) >>> 950 263099 273968 0 15004 14427 /usr/src/sys/vm/uma_core.c:2450 (512) >>> 108 150422 175840 0 10978 22988519 /usr/src/sys/vm/uma_core.c:1888 >>> (mbuf) >> >> > Here you can see that high UMA activity happens in periods of low disk >> > performance. But I'm not sure whether this is a root of the >> problem, not >> > a consequence. >> >> The extremely high contention there does seem to say you have a mbuf >> starvation problem and not a disk problem. I don't know why this >> would be happening off-hand. > But there's no mbuf shortage in `netstat -m`. > > What else can I try to track down the source of the problem? > >> Can you also provide more details about the system hardware and >> configuration? > This is Dell 2850 2 x Xeon 3.2, 4Gb RAM, 6x300Gb SCSI RAID5. I'll attach > details. > > With best regards, > Alexey Popov > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" last pid: 11008; load averages: 0.07, 0.10, 0.08 up 47+08:32:50 11:46:15 38 processes: 1 running, 37 sleeping Mem: 46M Active, 3443M Inact, 246M Wired, 144M Cache, 208M Buf, 5596K Free Swap: 2048M Total, 4K Used, 2048M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56386 root 1 4 0 19856K 10000K kqread 1 115:19 2.88% lighttpd 636 root 1 96 0 18292K 4212K select 0 25:39 0.00% snmpd 784 root 1 96 0 19668K 2072K select 1 2:31 0.00% sshd 680 root 1 96 0 7732K 1384K select 0 1:59 0.00% ntpd 1540 root 1 96 0 35092K 6496K select 0 1:30 0.00% httpd 769 root 4 20 0 14148K 2632K kserel 0 1:04 0.00% bacula-fd 755 root 1 96 0 3852K 1060K select 1 0:22 0.00% master 568 root 1 96 0 3648K 908K select 0 0:18 0.00% syslogd 80663 root 1 8 0 3688K 1016K nanslp 1 0:05 0.00% cron 760 postfix 1 96 0 3944K 1160K select 0 0:04 0.00% qmgr 89776 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89763 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89774 www 1 20 0 35180K 6684K lockf 0 0:04 0.00% httpd 89775 www 1 96 0 35180K 6684K select 0 0:04 0.00% httpd 699 root 1 20 0 7732K 1388K pause 0 0:03 0.00% ntpd 484 root 1 4 0 652K 220K select 0 0:00 0.00% devd 10904 llp 1 96 0 30616K 3564K select 0 0:00 0.00% sshd 10915 root 1 20 0 3912K 2340K pause 1 0:00 0.00% csh You run apache with mod_perl or php too. How many clients handle this apache server? Also in this light load you have locked files! Check script execution times (/server-status may be useful). When you have hight load check swap usage and haw many processes are in lockf state. Best Regards -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHFHuVxJBWvpalMpkRAnqTAJ9FgURNk98dtD0HYX6xIz17R6sLpQCgh5nJ XBtfOyzJJbkjzVzSF/WfmHc= =oTHZ -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 11:23:48 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C90216A418; Tue, 16 Oct 2007 11:23:48 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id 2B4D313C467; Tue, 16 Oct 2007 11:23:45 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.13) with ESMTPA id 194370688; Tue, 16 Oct 2007 15:21:06 +0400 Message-ID: <47149E6E.9000500@chistydom.ru> Date: Tue, 16 Oct 2007 15:20:14 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> In-Reply-To: <47147E49.9020301@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 16 Oct 2007 11:27:11 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 11:23:48 -0000 Hi. Kris Kennaway wrote: >>>> After some time of running under high load disk performance become >>>> expremely poor. At that periods 'systat -vm 1' shows something like >>>> this: >>> What does "high load" mean? You need to explain the system workload >>> more. >> This web service is similiar to YouTube. This server is video store. I >> have around 200G of *.flv (flash video) files on the server. >> >> I run lighttpd as a web server. Disk load is usually around 50%, network >> output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. >> >> As you can see it is a trivial service - sending files to network via >> HTTP. > Does lighttpd actually use HTTP accept filters? Don't know how to make sure, but is seems to run appropriate setsockopt (truss output): setsockopt(0x4,0xffff,0x1000,0x7fffffffe620,0x100) = 0 (0x0) > Are you using ipfilter and ipfw? You are paying a performance penalty > for having them. I'm using ipfw and one of the first rules is to pass all TCP established. ipfilter is not used on this server, but it is present in kernel as it can be used on other servers. I have 95% CPU idle, so I think packet filters does not produce significant load on this server. > You might try increasing BUCKET_MAX in sys/vm/uma_core.c. I don't > really understand the code here, but you seem to be hitting a threshold > behaviour where you are constantly running out of space in the per CPU > caches. Thanks, I'll try this. > This can happen if your workload is unbalanced between the CPUs and you > are always allocating on one but freeing on another, but I wouldn't > expect it should happen on your workload. Maybe it can also happen if > your turnover is high enough. This is very unlikely, because I have 5 another video storage servers of the same hardware and software configurations and they feel good. On the other side, all other servers were put in production before or after problematic servers and were filled with content in the other ways and therefore they could have slightly differerent load pattern. Totally I faced this bug three times: 1. The first time there was AFAIR 5.4-RELEASE on DELL 2850 with the same configuration as now. It was mp3 store and I used thttpd as HTTP server to serve mp3's. That time the problems were not so frequent and also it took too long to get back to normal operation so we had to reboot servers once a week or so. The problems began when we moved to new hardware - Dell 2850. That time we suspected amrd driver and had no time to dig in, bacause all the servers of the project were problematic. Installing Linux helped. 2. The second time it was server for static files of the very popular blog. The http server was nginx and disk contented puctures, mp3's and videos. It was Dell 1850 2x146 SCSI mirror. Linux also solved the problem. 3. The problem we see now. At first glance one can say that problem is in Dell's x850 series or amr(4), but we run this hardware on many other projects and they work well. Also Linux on them works. And few hours ago I received feed back from Andrzej Tobola, he has the same problem on FreeBSD 7 with Promise ATA software mirror: === Subject: Re: amrd disk performance drop after running under high load Date: Tue, 16 Oct 2007 10:59:34 +0200 From: Andrzej Tobola To: Alexey Popov Exactly the same here but on big ata RAID0 with big trafic (~10GB/24h): amper% df -h /ftp/priv Filesystem Size Used Avail Capacity Mounted one /dev/ar0a 744G 679G 4.7G 99% /ftp/priv amper% grep ^ar /var/run/dmesg.boot ar0: 763108MB status: READY ar0: disk0 READY using ad6 at ata3-master ar0: disk1 READY using ad4 at ata2-master amper% uname -a FreeBSD xxx 7.0-CURRENT-200709 FreeBSD 7.0-CURRENT-200709 #0: Tue Sep 11 04:44:48 UTC 2007 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 I am rebooting if I reach this state (approx. a week). It is old bug - a few months ;) cheers, -a === So I can conclude that FreeBSD has a long standing bug in VM that could be triggered when serving large amount of static data (much bigger than memory size) on high rates. Possibly this only applies to large files like mp3 or video. > What does vmstat -z show during the good and bad times? I'll send this data when the bad times will happen next time. With best regards, Alexey Popov From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 11:30:47 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FDEA16A41A for ; Tue, 16 Oct 2007 11:30:46 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4D88913C46A for ; Tue, 16 Oct 2007 11:30:46 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 086BD1CC065; Tue, 16 Oct 2007 04:30:46 -0700 (PDT) Date: Tue, 16 Oct 2007 04:30:46 -0700 From: Jeremy Chadwick To: freebsd-hackers@freebsd.org Message-ID: <20071016113046.GA35318@eos.sc1.parodius.com> Mail-Followup-To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.16 (2007-06-09) Subject: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 11:30:47 -0000 Since the snapshot code (e.g. mksnap_ffs(8) and friends) was introduced, dump(8) was modified to nag you if you didn't use the -L argument. "Um, okay, I'd better use -L" is what came out of my mouth, and I'm sure a lot of other administrators' when they saw this message. But it seems the making a snapshot is an incredibly slow/intensive task. The documentation I've read indicates that making a snapshot "is incredibly fast" -- based on my experiences, it isn't. At least it's no where near as fast as, say, a Netapp filer. I've found 3 threads (dating 2003, 2005, and 2007) about this problem: http://lists.freebsd.org/pipermail/freebsd-current/2003-August/009135.html http://lists.freebsd.org/pipermail/freebsd-fs/2005-July/001216.html http://lists.freebsd.org/pipermail/freebsd-stable/2007-January/031882.html This issue is still present on RELENG_7, and I can confirm it on multiple machines (some running *completely* different hardware than others). osiris# df -ki /disk2 Filesystem 1024-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad6s1d 236511738 4 217590796 0% 2 30570492 0% /disk2 osiris# time mksnap_ffs /disk2 /disk2/mysnapshot 0.000u 1.012s 5:12.23 0.3% 5+1149k 7803+18819io 0pf+0w While mksnap_ffs runs, the process remains in wdrain state. gstat(8) shows immense disk I/O. ms/r occasionally jumps up to 1100 or higher, but usually hovers around 40-60. osiris# gstat -I500ms -f'ad6' dT: 0.501s w: 0.500s filter: ad6 L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 2 80 52 830 38.6 28 447 22.4 100.2| ad6 2 80 52 830 38.6 28 447 22.4 100.2| ad6s1 0 0 0 0 0.0 0 0 0.0 0.0| ad6s1c 2 80 52 830 38.6 28 447 22.4 100.2| ad6s1d Now for snapshot removal: osiris# time rm /disk2/mysnapshot override r--r----- root/operator snapshot for /disk2/mysnapshot? y 0.000u 0.285s 1:58.03 0.2% 16+1161k 7456+7456io 0pf+0w While rm runs, the process remains in biord state. During either of these operations, the system can occasionally go into a "stalled" state, where any disk operations remain deadlocked until the mksnap_ffs or rm are finished. I ran a second mksnap_ffs "just to see" what happened. Between the first time and this time, *nothing* happened on the filesystem (no disk reads or writes AFAIK): osiris# time mksnap_ffs /disk2 /disk2/mysnapshot 0.016u 1.352s 10:13.73 0.2% 5+1164k 14501+27931io 0pf+0w The time doubled. This isn't good. Disks are getting larger, filesystems growing, people storing more data. Hitachi, for example, has guaranteed 4TB disks by the end of 2011. If this problem has sat idle for at least 4 years already, we'll be in a lot of trouble come 2011. And let's not forget that every piece of FreeBSD documentation tells admins to "use dump, it's the best!". This issue is a good reason to consider using tools like rsync or tar instead. :-( I will gladly work with anyone who wishes to tackle this, either by providing hardware (MB/disks/etc.) for free, or by giving the individual access to a box that has serial console + a serial debugger available. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 11:54:23 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 047FD16A41A for ; Tue, 16 Oct 2007 11:54:23 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 025F613C45B for ; Tue, 16 Oct 2007 11:54:22 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l9GBsJxp055823 for ; Tue, 16 Oct 2007 06:54:21 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <4714A663.5010800@freebsd.org> Date: Tue, 16 Oct 2007 06:54:11 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <20071016113046.GA35318@eos.sc1.parodius.com> In-Reply-To: <20071016113046.GA35318@eos.sc1.parodius.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 11:54:23 -0000 Jeremy Chadwick wrote: > Since the snapshot code (e.g. mksnap_ffs(8) and friends) was introduced, > dump(8) was modified to nag you if you didn't use the -L argument. "Um, > okay, I'd better use -L" is what came out of my mouth, and I'm sure a > lot of other administrators' when they saw this message. > > But it seems the making a snapshot is an incredibly slow/intensive task. > The documentation I've read indicates that making a snapshot "is > incredibly fast" -- based on my experiences, it isn't. At least it's no > where near as fast as, say, a Netapp filer. The problem is the way the snapshots work in UFS2. It has to do a lot of work to create that snapshot, and the amount of work it does goes up with the amount space you have available (because it relates to the number of cylinder groups you have). The UFS2 snapshot and the WAFL (NetApp's file system) snapshot are *completely* different, and should not be compared in this way. The functionality is (in the end) the same, but otherwise, they are different. > I've found 3 threads (dating 2003, 2005, and 2007) about this problem: > > http://lists.freebsd.org/pipermail/freebsd-current/2003-August/009135.html > http://lists.freebsd.org/pipermail/freebsd-fs/2005-July/001216.html > http://lists.freebsd.org/pipermail/freebsd-stable/2007-January/031882.html Only three threads? :) There's probably hundreds like them.. > This issue is still present on RELENG_7, and I can confirm it on > multiple machines (some running *completely* different hardware than > others). It's a UFS2 problem, and the docs that say 'incredibly fast' are actually referring to small filesystems, that are not busy (with writes). Maybe the docs should be clarified for now. You can submit patches to the docs you found that say that if you'd like to help out. > osiris# df -ki /disk2 > Filesystem 1024-blocks Used Avail Capacity iused ifree %iused Mounted on > /dev/ad6s1d 236511738 4 217590796 0% 2 30570492 0% /disk2 > > osiris# time mksnap_ffs /disk2 /disk2/mysnapshot > 0.000u 1.012s 5:12.23 0.3% 5+1149k 7803+18819io 0pf+0w > > While mksnap_ffs runs, the process remains in wdrain state. gstat(8) > shows immense disk I/O. ms/r occasionally jumps up to 1100 or higher, > but usually hovers around 40-60. [..snip..] > The time doubled. This isn't good. > > Disks are getting larger, filesystems growing, people storing more data. > Hitachi, for example, has guaranteed 4TB disks by the end of 2011. If > this problem has sat idle for at least 4 years already, we'll be in a > lot of trouble come 2011. And let's not forget that every piece of > FreeBSD documentation tells admins to "use dump, it's the best!". This > issue is a good reason to consider using tools like rsync or tar > instead. :-( I recommend reading up a little bit on how the snapshots for UFS2 work. It will give you a good understanding of what the issue is. Essentially, your disk is hammered making copies of all the cylinder groups, skipping those that are 'busy', and coming back to them later. On a 200Gb disk, you could have 1000 cylinder groups, each having to be locked, copied, unlocked, and then checked again for any subsequent changes. The stalls you see are when there are lock contentions, or disk IO issues. On a single disk (like your setup above), your snapshots will take forever since there is very little random IO performance available to you. > I will gladly work with anyone who wishes to tackle this, either by > providing hardware (MB/disks/etc.) for free, or by giving the individual > access to a box that has serial console + a serial debugger available. FreeBSD 7 includes ZFS. Have you thought about using it? The problem isn't that developers don't know the problem exists, or that they don't have hardware, or a serial console access to a system. The problem is that there are only so many developers, and so much time, and this is a big mountain to climb. It's hard to find an experienced person to do the work (for free), when they could be doing anything else they wish. I think, that in the end, for some of these aging issues to get resolved, there needs to be another bounty put out on it. I think rsync.net might even have one started for this issue already - you might think about adding to the bounty, or officially offering hardware through there. Eric From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 12:24:49 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2BD1E16A418 for ; Tue, 16 Oct 2007 12:24:49 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 2B87D13C459 for ; Tue, 16 Oct 2007 12:24:49 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id EA78D1CC02E; Tue, 16 Oct 2007 05:24:48 -0700 (PDT) Date: Tue, 16 Oct 2007 05:24:48 -0700 From: Jeremy Chadwick To: Eric Anderson Message-ID: <20071016122448.GA36973@eos.sc1.parodius.com> Mail-Followup-To: Eric Anderson , freebsd-hackers@freebsd.org References: <20071016113046.GA35318@eos.sc1.parodius.com> <4714A663.5010800@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4714A663.5010800@freebsd.org> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 12:24:49 -0000 On Tue, Oct 16, 2007 at 06:54:11AM -0500, Eric Anderson wrote: > The problem is the way the snapshots work in UFS2. [...] > ... the docs that say 'incredibly fast' are actually referring to > small filesystems, that are not busy (with writes). Maybe the docs > should be clarified for now. You can submit patches to the docs you > found that say that if you'd like to help out. I'll work on that this week. I'm not sure if I read it in the FreeBSD Handbook or the FAQ or someplace on the web. If it's in the lesser two, I'll send-pr changes. > I recommend reading up a little bit on how the snapshots for UFS2 > work. It will give you a good understanding of what the issue is. > Essentially, your disk is hammered making copies of all the cylinder > groups, skipping those that are 'busy', and coming back to them later. > On a 200Gb disk, you could have 1000 cylinder groups, each having to > be locked, copied, unlocked, and then checked again for any subsequent > changes. The stalls you see are when there are lock contentions, or > disk IO issues. On a single disk (like your setup above), your > snapshots will take forever since there is very little random IO > performance available to you. I understand the majority of what you've written, and this definitely gives me a better insight to what the problem is. I hope that others will eventually find this thread and find their answer here. Thank you for providing this. That said, for such setups, would you recommend *not* using snapshots? If so, possibly we should consider removing the following code from src/sbin/dump/main.c: 334 } else if (snapdump == 0) { 335 msg("WARNING: %s\n", 336 "should use -L when dumping live read-write " 337 "filesystems!"); 338 } else { > FreeBSD 7 includes ZFS. Have you thought about using it? I haven't. For starters, I keep seeing mails from users reporting data corruption or kernel panics when using it. This doesn't mean ZFS is "bad" (it's likely ZFS is exposing data corruption for them that's occuring at a lower level (controller, disk, or RAM)), but it does keep me from considering it a worthy alternative to UFS2 on production systems at this time. And then there's this: WARNING: ZFS is considered to be an experimental feature in FreeBSD. This doesn't give admins "warm fuzzies". :-) I'm considering trying ZFS on my home system (a ZFS filesystem atop a gstripe(8) pair), where I also perform nightly backups using dump, but there's a part of me which is asking "just how important is your data? What if ZFS breaks in the middle of you using dump(8) on it? What then?" > The problem isn't that developers don't know the problem exists, or > that they don't have hardware, or a serial console access to a system. > The problem is that there are only so many developers, and so much > time, and this is a big mountain to climb. It's hard to find an > experienced person to do the work (for free), when they could be doing > anything else they wish. I think, that in the end, for some of these > aging issues to get resolved, there needs to be another bounty put out > on it. I think rsync.net might even have one started for this issue > already - you might think about adding to the bounty, or officially > offering hardware through there. I'll take a peek at the site. I have no problem donating some money and/or hardware to the cause, but as I'm one person, I don't have an infinite supply of cash to donate to projects like this. About the most I could donate would be US$1000 (and that's a bit extreme), which wouldn't provide for very much by itself. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 14:37:31 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45BA216A418 for ; Tue, 16 Oct 2007 14:37:31 +0000 (UTC) (envelope-from BearPerson@gmx.net) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.freebsd.org (Postfix) with SMTP id A14FE13C480 for ; Tue, 16 Oct 2007 14:37:30 +0000 (UTC) (envelope-from BearPerson@gmx.net) Received: (qmail invoked by alias); 16 Oct 2007 14:10:49 -0000 Received: from port-83-236-56-222.dynamic.qsc.de (EHLO 39-25.mops.rwth-aachen.de) [83.236.56.222] by mail.gmx.net (mp029) with SMTP; 16 Oct 2007 16:10:49 +0200 X-Authenticated: #20254835 X-Provags-ID: V01U2FsdGVkX1+uuyxoHIyui9kI7cdiIsTpZ68kSYxqouJYOXMfoC A0FAPYEctXGfM5 Date: Tue, 16 Oct 2007 16:10:37 +0200 From: Karsten Behrmann To: freebsd-hackers@freebsd.org Message-ID: <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> In-Reply-To: <20071011022001.GC13480@gandalf.sssup.it> References: <20071011022001.GC13480@gandalf.sssup.it> Organization: RWTH Aachen X-Mailer: Claws Mail 2.10.0 (GTK+ 2.6.10; i386-apple-darwin8.10.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 14:37:31 -0000 > Hi, > is anybody working on the `Pluggable Disk Scheduler Project' from > the ideas page? I've been kicking the idea around in my head, but I'm probably newer to everything involved than you are, so feel free to pick it up. If you want, we can toss some ideas and code to each other, though I don't really have anything on the latter. [...] > After reading [1], [2] and its follow-ups the main problems that > need to be addressed seem to be: > > o is working on disk scheduling worth at all? Probably, one of the main applications would be to make the background fsck a little more well-behaved. > o Where is the right place (in GEOM) for a disk scheduler? I have spent some time at eurobsdcon talking to Kirk and phk about this, and the result was that I now know strong proponents both for putting it into the disk drivers and for putting it into geom ;-) Personally, I would put it into geom. I'll go into more detail on this later, but basically, geom seems a better fit for "high-level" code than a device driver, and if done properly performance penalties should be negligible. > o How can anticipation be introduced into the GEOM framework? I wouldn't focus on just anticipation, but also other types of schedulers (I/O scheduling influenced by nice value?) > o What can be an interface for disk schedulers? good question, but geom seems a good start ;) > o How to deal with devices that handle multiple request per time? Bad news first: this is most disks out there, in a way ;) SCSI has tagged queuing, ATA has native command queing or whatever the ata people came up over their morning coffee today. I'll mention a bit more about this further down. > o How to deal with metadata requests and other VFS issues? Like any other disk request, though for priority-respecting schedulers this may get rather interesting. [...] > The main idea is to allow the scheduler to enqueue the requests > having only one (other small fixed numbers can be better on some > hardware) outstanding request and to pass new requests to its > provider only after the service of the previous one ended. You'll want to queue at least two requests at once. The reason for this is performance: Currently, drivers queue their own I/O. This means that as soon as a request completes (on devices that don't have in-device queues), they can fairly quickly grab a new request from their internal queue and push it back to the device from the interrupt handler or some other fast method. Having the device idle while the response percolates up the geom stack and a new request down will likely be rather wasteful. For disks with queuing, I'd recommend trying to keep the queue reasonably full (unless the queuing strategy says otherwise), for disks without queuing I'd say we want to push at least one more request down. Personally, I think the sanest design would be to have device drivers return a "temporary I/O error" along the lines of EAGAIN, meaning their queue is full. > The example scheduler in the draft takes the following approach: > > o a scheduling GEOM class is introduced. It can be stacked on > top of disk geoms, and schedules all the requests coming > from its consumers. I'm not absolutely sure that a new class > is really needed but I think that it can simplify testing and > experimenting with various solutions on the scheduler placement. Probably, though we'll want to make sure that they stack on top of (or are inside of?) the geoms talking to the disks, because it rarely makes sense to put a queuing geom on top of, say, a disklabel geom. The advantage of making it a full geom is configurability. You would be able to swap out a scheduler at runtime, select different sched- ulers for different disks, and potentially even test new schedulers without rebooting (though you wouldn't want to do that for benchmarks) > o Requests coming from consumers are passed down immediately > if there is no other request under service, otherwise they > are queued in a bioq. This is specific to the anticipatory scheduler. I would say in more general terms: - A queuing geom is to push all requests that it wants serviced down towards the disk, until the disk reports a queue full. A queuing geom is allowed to hold back requests even when the driver queue is not full yet, if it does not want the disk to attempt such I/O yet (such as the anticipatory scheduler waiting for another disk request near the last one, or the process-priority scheduler holding back a low- priority request that would potentially cause a long seek, until io has been idle) This dispels phk's anti-geom argument of "it will be inefficient because it will take longer for a new request to get to the driver" - if the queuing strategy had wanted the request to be sent to the drive, it would already have sent it. (The exception is that the disk will have its internal queue a little more empty than it could be - not a big issue with queue sizes of 8 or 16) [...] > So, as I've said, I'd like to know what you think about the subject, > if I'm missing something, if there is some kind of interest on this > and if/how can this work proceed. I would think that this would be quite useful, but I don't think my voice counts for much ;-) It would help - servers where anticipatory performs better than elevator - realtime environments that need a scheduler fitting their needs - the background fsck, if someone implements a "priority" scheduler Anyway, that's a quick dump of my thoughts on the subject so far, I've myself wanted to get started on this but didn't get around to it yet (I'm fairly new to FreeBSD). If you want to hash some ideas out with me, I'll be watching my inbox, this ML, and you can reach me on IRC as BearPerson on freenode, quakenet, undernet, or whatever else you ask me to connect to, or via whatever other method convenient to you ;) So Far, Karsten "BearPerson" Behrmann From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 15:05:50 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EFD816A419 for ; Tue, 16 Oct 2007 15:05:50 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 98E4613C4B0 for ; Tue, 16 Oct 2007 15:05:50 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 2DC021CC030; Tue, 16 Oct 2007 08:05:50 -0700 (PDT) Date: Tue, 16 Oct 2007 08:05:50 -0700 From: Jeremy Chadwick To: Simun Mikecin Message-ID: <20071016150550.GA40548@eos.sc1.parodius.com> Mail-Followup-To: Simun Mikecin , freebsd-hackers@freebsd.org References: <5949.42099.qm@web36608.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5949.42099.qm@web36608.mail.mud.yahoo.com> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 15:05:50 -0000 On Tue, Oct 16, 2007 at 06:44:36AM -0700, Simun Mikecin wrote: > Not using a snapshot for dump may produce inconsistent dump image if there was writing during > dumping. Maybe it should say something like "should use -L when dumping live read-write > filesystems for the result to be consistent (at the cost of speed)!". But that is too long :( I thought that the only way you could get an inconsistent filesystem dump was if you used the -C (caching) option in dump? I tried removing -L from my (home) backups, and I found that it made a world of difference (speed-wise) on all of my filesystems except for /storage (see thread). Without -L, dump estimated 2 hours 10 minutes; with -L, it estimated 50 minutes. This is a little odd (the inconsistencies in speed/time), but I can't argue with the results. The fact of the matter is, people always want a consistent (that is to say, working) backup. So if using -L is the proper solution regardless of the UFS2 drawbacks, then I'll have to live with that. Or go to ZFS -- more on that below. > One of great things about ZFS is that you can forget about things like gstripe(8) or dump(8). You > only need two commands: zpool and zfs. ZFS is not just a filesystem, it's also a logical volume > management tool. > > ZFS on FreeBSD is considered experimental since it is very new. But from experience so far with > it, only a few glitches do still exist: > 1) root on ZFS is possible, but it could give you more problems then it solves (for now, it's best > to have a small, say 512MB root filesystem running UFS, but everything else on ZFS). > 2) using a zvol on ZFS for swap can cause a panic > 3) using ZFS on FreeBSD/i386 can cause a panic (I suggest using UFS+gjournal instead of ZFS on > FreeBSD/i386) In regards to the items you list: 1) doesn't apply (I have no issues with using UFS or UFS on rootfs), 2) not familiar with this feature of ZFS, and 3) would definitely impact us, as we use i386 exclusively. Now, that said... There's a current thread discussing system panics (on i386 and amd64!) with ZFS (re: "ZFS kmem_map too small"). This is one of the threads I'm referring to. > Personally, I would choose ZFS on FreeBSD/amd64 production machine. None of the systems here contain more than 2GB of RAM, and generally-speaking won't benefit from any of the bonuses amd64 offers at this time. There's other scenarios here that don't permit me to run amd64 on our production boxes (has nothing to do with FreeBSD), so for now, we stick with i386. On the bright side, we do now have a machine running RELENG_7 (I installed the box this weekend), but the two requirements for this box are 1) that the machine remain up and responsive as close to 24x7 as possible (e.g. stalling disk I/O like dump -L on large UFS2 filesystems isn't acceptable), 2) remains stable, and 3) runs i386 (developer is not familiar with 64-bit environments). I'll have to discuss with the developer if he feels comfortable with ZFS there. On my home machine, I'm more than willing to run amd64 -- and I have in the past (but went back to i386 because I did not feel comfortable with things like /usr/lib32; discuss off-list if interested) -- but my requirements are a bit different. In the case of my home machine, I spent a little time this morning migrating it from UFS2/gstripe (the /storage filesystem consists of two SATA300 disks in a RAID-0 array -- yes, you read that right, hence the nightly backups!) to a ZFS storage pool (zpool). Filesystem 1024-blocks Used Avail Capacity Mounted on storage 957873408 94645376 863228032 10% /storage So far so good; and I wanted to try scrubbing, just for fun... icarus# zpool status pool: storage state: ONLINE scrub: scrub in progress, 49.74% done, 0h6m to go config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 errors: No known data errors I'll have to see how ZFS snapshots work out. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 14:11:19 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8254016A41A for ; Tue, 16 Oct 2007 14:11:19 +0000 (UTC) (envelope-from numisemis@yahoo.com) Received: from web36608.mail.mud.yahoo.com (web36608.mail.mud.yahoo.com [209.191.85.25]) by mx1.freebsd.org (Postfix) with SMTP id 5CE6A13C468 for ; Tue, 16 Oct 2007 14:11:19 +0000 (UTC) (envelope-from numisemis@yahoo.com) Received: (qmail 42458 invoked by uid 60001); 16 Oct 2007 13:44:37 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=b6l9ZYL2wHtq8yl/N1vf/7yTrBsycam3dJyQ1cQ2MPZa8Balfe961pGRmiGD02Qz9ELXC9OwmN9Om8T4CqAZt4eqanXPcPH7oJDGVa4U5FqnCHq6MtSKUhwGHvUAHtmqf161s724yH3CALwknpnf5YyZhrLiBkM2zZ1vneOCsy4=; X-YMail-OSG: 956Nq1sVM1l3majznyd5geEAuVJdc9DzG1MXG92mLnCqEB3tTGt3Yjj8kMvlPylTrr2_cmzlOkt7HU0TmpYKtaaJOZOwXAUIT167gbx2F_LspFIbiU5ey7HBNzp7ug-- Received: from [213.147.110.159] by web36608.mail.mud.yahoo.com via HTTP; Tue, 16 Oct 2007 06:44:36 PDT Date: Tue, 16 Oct 2007 06:44:36 -0700 (PDT) From: Simun Mikecin To: Jeremy Chadwick MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <5949.42099.qm@web36608.mail.mud.yahoo.com> X-Mailman-Approved-At: Tue, 16 Oct 2007 15:38:10 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 14:11:19 -0000 Jeremy Chadwich wrote: >That said, for such setups, would you recommend *not* using snapshots? >If so, possibly we should consider removing the following code from >src/sbin/dump/main.c: >334 } else if (snapdump == 0) { >335 msg("WARNING: %s\n", >336 "should use -L when dumping live read-write " >337 "filesystems!"); >338 } else { Not using a snapshot for dump may produce inconsistent dump image if there was writing during dumping. Maybe it should say something like "should use -L when dumping live read-write filesystems for the result to be consistent (at the cost of speed)!". But that is too long :( >> FreeBSD 7 includes ZFS. Have you thought about using it? >I haven't. For starters, I keep seeing mails from users reporting data >corruption or kernel panics when using it. This doesn't mean ZFS is >"bad" (it's likely ZFS is exposing data corruption for them that's >occuring at a lower level (controller, disk, or RAM)), but it does keep >me from considering it a worthy alternative to UFS2 on production >systems at this time. And then there's this: >WARNING: ZFS is considered to be an experimental feature in FreeBSD. >This doesn't give admins "warm fuzzies". :-) I'm considering trying ZFS >on my home system (a ZFS filesystem atop a gstripe(8) pair), where I >also perform nightly backups using dump, but there's a part of me which >is asking "just how important is your data? What if ZFS breaks in the >middle of you using dump(8) on it? What then?" One of great things about ZFS is that you can forget about things like gstripe(8) or dump(8). You only need two commands: zpool and zfs. ZFS is not just a filesystem, it's also a logical volume management tool. ZFS on FreeBSD is considered experimental since it is very new. But from experience so far with it, only a few glitches do still exist: 1) root on ZFS is possible, but it could give you more problems then it solves (for now, it's best to have a small, say 512MB root filesystem running UFS, but everything else on ZFS). 2) using a zvol on ZFS for swap can cause a panic 3) using ZFS on FreeBSD/i386 can cause a panic (I suggest using UFS+gjournal instead of ZFS on FreeBSD/i386) Personally, I would choose ZFS on FreeBSD/amd64 production machine. ____________________________________________________________________________________ Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=list&sid=396545469 From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 17:14:19 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 111A816A417 for ; Tue, 16 Oct 2007 17:14:19 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from spork.qfe3.net (spork.qfe3.net [212.13.207.101]) by mx1.freebsd.org (Postfix) with ESMTP id CBFB313C46A for ; Tue, 16 Oct 2007 17:14:18 +0000 (UTC) (envelope-from tom.hurst@clara.net) Received: from [81.104.144.87] (helo=voi.aagh.net) by spork.qfe3.net with esmtp (Exim 4.66 (FreeBSD)) (envelope-from ) id 1Ihpki-0000TC-Gp; Tue, 16 Oct 2007 17:58:44 +0100 Received: from freaky by voi.aagh.net with local (Exim 4.67 (FreeBSD)) (envelope-from ) id 1Ihpki-0001yS-9P; Tue, 16 Oct 2007 17:58:44 +0100 Date: Tue, 16 Oct 2007 17:58:44 +0100 From: Thomas Hurst To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org Message-ID: <20071016165844.GA6566@voi.aagh.net> Mail-Followup-To: freebsd-stable@freebsd.org, freebsd-hackers@freebsd.org References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47149E6E.9000500@chistydom.ru> Organization: Not much. User-Agent: Mutt/1.5.16 (2007-06-09) Sender: Thomas Hurst Cc: Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 17:14:19 -0000 * Alexey Popov (lol@chistydom.ru) wrote: > So I can conclude that FreeBSD has a long standing bug in VM that > could be triggered when serving large amount of static data (much > bigger than memory size) on high rates. Possibly this only applies to > large files like mp3 or video. I've seen highly dubious VM behavior when reading large files locally; the system ends up swapping out a small but significant amount of various processes, even very small recently active ones like syslogd, for no apparant reason: http://lists.freebsd.org/pipermail/freebsd-stable/2007-September/036956.html I've also seen dubious IO behavior from amr(4), where access to one array will interfere with IO from an independent set of spindles that just happen to be attached to the same card: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/114438 Given the blank looks I've received every time I've mentioned these things I'm guessing they aren't seen by others all that often, but maybe one or both are vaguely relevent to your situation. -- Thomas 'Freaky' Hurst http://hur.st/ From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 18:11:25 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5780516A417 for ; Tue, 16 Oct 2007 18:11:25 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from canonware.com (canonware.com [64.183.146.166]) by mx1.freebsd.org (Postfix) with ESMTP id 3DC2513C46B for ; Tue, 16 Oct 2007 18:11:24 +0000 (UTC) (envelope-from jasone@freebsd.org) Received: from [192.168.168.201] (canonware.com [64.183.146.166]) by canonware.com (Postfix) with ESMTP id B597F1298E3 for ; Tue, 16 Oct 2007 10:55:42 -0700 (PDT) Message-ID: <4714F8A2.5050406@freebsd.org> Date: Tue, 16 Oct 2007 10:45:06 -0700 From: Jason Evans User-Agent: Thunderbird 1.5.0.12 (X11/20070718) MIME-Version: 1.0 To: FreeBSD Hackers References: <20071016094118.GE5411@hoeg.nl> In-Reply-To: <20071016094118.GE5411@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 18:11:25 -0000 Ed Schouten wrote: > For some reason, I want to understand how the queueing of blocked > threads in the kernel works when waiting for a lock, which is if I > understand correctly done by the turnstiles and sleepqueues. I'm the > proud owner of The Design and Implementation of the FreeBSD Operating > System book, but for some reason, I can't find anything about it in the > book. > > Is there a way to obtain information about how they work? I already read > the source somewhat, but that shouldn't be an ideal solution, in my > opinion. You might take a look at _Solaris Internals_ by Mauro and McDougall. Jason From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 18:22:51 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38E6E16A419 for ; Tue, 16 Oct 2007 18:22:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outL.internet-mail-service.net (outL.internet-mail-service.net [216.240.47.235]) by mx1.freebsd.org (Postfix) with ESMTP id 2379613C43E for ; Tue, 16 Oct 2007 18:22:51 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Tue, 16 Oct 2007 11:22:50 -0700 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 4436E1266E6; Tue, 16 Oct 2007 11:22:48 -0700 (PDT) Message-ID: <47150189.90403@elischer.org> Date: Tue, 16 Oct 2007 11:23:05 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Jason Evans References: <20071016094118.GE5411@hoeg.nl> <4714F8A2.5050406@freebsd.org> In-Reply-To: <4714F8A2.5050406@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 18:22:51 -0000 Jason Evans wrote: > Ed Schouten wrote: >> For some reason, I want to understand how the queueing of blocked >> threads in the kernel works when waiting for a lock, which is if I >> understand correctly done by the turnstiles and sleepqueues. I'm the >> proud owner of The Design and Implementation of the FreeBSD Operating >> System book, but for some reason, I can't find anything about it in the >> book. >> >> Is there a way to obtain information about how they work? I already read >> the source somewhat, but that shouldn't be an ideal solution, in my sleepqueues and turnstiles are relatively new. they may have come in since 5.2 (which is what the book was based on I think). >> opinion. > > You might take a look at _Solaris Internals_ by Mauro and McDougall. > > Jason > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 18:30:56 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED07B16A418; Tue, 16 Oct 2007 18:30:56 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id CC1BF13C46A; Tue, 16 Oct 2007 18:30:55 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4715035D.2090802@FreeBSD.org> Date: Tue, 16 Oct 2007 20:30:53 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Alexey Popov References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> In-Reply-To: <47149E6E.9000500@chistydom.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 18:30:57 -0000 Alexey Popov wrote: > Hi. > > Kris Kennaway wrote: > >>>>> After some time of running under high load disk performance become >>>>> expremely poor. At that periods 'systat -vm 1' shows something like >>>>> this: >>>> What does "high load" mean? You need to explain the system workload >>>> more. >>> This web service is similiar to YouTube. This server is video store. I >>> have around 200G of *.flv (flash video) files on the server. >>> >>> I run lighttpd as a web server. Disk load is usually around 50%, network >>> output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. >>> >>> As you can see it is a trivial service - sending files to network via >>> HTTP. >> Does lighttpd actually use HTTP accept filters? > Don't know how to make sure, but is seems to run appropriate setsockopt > (truss output): > > setsockopt(0x4,0xffff,0x1000,0x7fffffffe620,0x100) = 0 (0x0) OK. >> Are you using ipfilter and ipfw? You are paying a performance penalty >> for having them. > I'm using ipfw and one of the first rules is to pass all TCP > established. ipfilter is not used on this server, but it is present in > kernel as it can be used on other servers. I have 95% CPU idle, so I > think packet filters does not produce significant load on this server. Well, it was not your most serious issue, but from your profiling trace it is definitely burning cycles with every packet processed. >> You might try increasing BUCKET_MAX in sys/vm/uma_core.c. I don't >> really understand the code here, but you seem to be hitting a >> threshold behaviour where you are constantly running out of space in >> the per CPU caches. > Thanks, I'll try this. > >> This can happen if your workload is unbalanced between the CPUs and >> you are always allocating on one but freeing on another, but I >> wouldn't expect it should happen on your workload. Maybe it can also >> happen if your turnover is high enough. > This is very unlikely, because I have 5 another video storage servers of > the same hardware and software configurations and they feel good. Clearly something is different about them, though. If you can characterize exactly what that is then it will help. > On the other side, all other servers were put in production before or > after problematic servers and were filled with content in the other ways > and therefore they could have slightly differerent load pattern. > > Totally I faced this bug three times: > > 1. The first time there was AFAIR 5.4-RELEASE on DELL 2850 with the same > configuration as now. It was mp3 store and I used thttpd as HTTP server > to serve mp3's. That time the problems were not so frequent and also it > took too long to get back to normal operation so we had to reboot > servers once a week or so. > > The problems began when we moved to new hardware - Dell 2850. That time > we suspected amrd driver and had no time to dig in, bacause all the > servers of the project were problematic. Installing Linux helped. > > 2. The second time it was server for static files of the very popular > blog. The http server was nginx and disk contented puctures, mp3's and > videos. It was Dell 1850 2x146 SCSI mirror. Linux also solved the problem. > > 3. The problem we see now. > > At first glance one can say that problem is in Dell's x850 series or > amr(4), but we run this hardware on many other projects and they work > well. Also Linux on them works. OK but there is no evidence in what you posted so far that amr is involved in any way. There is convincing evidence that it is the mbuf issue. > And few hours ago I received feed back from Andrzej Tobola, he has the > same problem on FreeBSD 7 with Promise ATA software mirror: Well, he didnt provide any evidence yet that it is the same problem, so let's not become confused by feelings :) > So I can conclude that FreeBSD has a long standing bug in VM that could > be triggered when serving large amount of static data (much bigger than > memory size) on high rates. Possibly this only applies to large files > like mp3 or video. It is possible, we have further work to do to conclude this though. Kris From owner-freebsd-hackers@FreeBSD.ORG Tue Oct 16 19:08:37 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5EE616A41B for ; Tue, 16 Oct 2007 19:08:37 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: from malasada.lava.net (malasada.lava.net [64.65.64.17]) by mx1.freebsd.org (Postfix) with ESMTP id 8696D13C45A for ; Tue, 16 Oct 2007 19:08:37 +0000 (UTC) (envelope-from cliftonr@lava.net) Received: by malasada.lava.net (Postfix, from userid 102) id 7D84E153882; Tue, 16 Oct 2007 09:08:36 -1000 (HST) Date: Tue, 16 Oct 2007 09:08:36 -1000 From: Clifton Royston To: FreeBSD hackers list Message-ID: <20071016190834.GC17120@lava.net> Mail-Followup-To: FreeBSD hackers list Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Subject: Re: Creating install CD with custom ports - how to massage INDEX file? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2007 19:08:37 -0000 [Sorry, References: header lost] Peter Jeremy wrote: > In which case, you should be able to "cd /usr/ports && make index" I can see a few points of confusion here, so let me clarify: 1) As I said, it's under my own CVS tree, independent of the system ports, so the corresponding step is "cd ~/sandbox/ports && make index". That part works fine as of late, that's how I got to understand the "dependency closure" bit. 2) There are a couple funky steps involved right after that, because although the ports system includes a PORTSDIR makefile variable, AFAICT the index-building appears to not fully accept it, so I have to massage out references to "/home/cliftonr/sandbox/ports/foo" in the INDEX-6 file that comes out. Not a huge problem, just a few ugly perl lines in my Makefile. Now the tricky bits: 3) The INDEX-6 file built in ports is not actually the same format as the INDEX file on an install CD. They're close, but it appears that some of the fields are swapped around and an extra field added at the end. If it only meant appending a "|1" to the end of each line, meaning everything's on disc 1, that would be easy, but as I say there appears to be some change in the field order and I'm having to guess what each field means. 4) I'm not currently trying to get it inside the chroot for the release build process, because that whole process is complex and rigid enough to be scary. To be more specific, it does its own complete CVS checkout from the local FreeBSD CVS repository for whatever tags you give it; if I were to enable ports/packages building, it appears to me it would likewise checkout the ports from the FreeBSD repository as part of "make release", rather than get them from our own repository. 5) If there's a window or stage where I could halt the release build, copy the desired ports tree in there, and then give a comand to say "continue, build the packages and index and slap them into the disc image", then that's exactly the set of commands I'm looking for. Whatever that command is, however, I believe it will either include the chrooted "cd /usr/ports && make index", or come after it. Someone pointed me at a particular .py script beneath /usr/src/release, so I need to go look at that, preferably after a bit more sleep and/or coffee. -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 08:20:42 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AEEA16A41B; Wed, 17 Oct 2007 08:20:41 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id 6129913C457; Wed, 17 Oct 2007 08:20:39 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4715C5D7.7060806@FreeBSD.org> Date: Wed, 17 Oct 2007 10:20:39 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Alexey Popov References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> In-Reply-To: <4715C297.1020905@chistydom.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 08:20:42 -0000 Alexey Popov wrote: >>> This is very unlikely, because I have 5 another video storage servers >>> of the same hardware and software configurations and they feel good. >> Clearly something is different about them, though. If you can >> characterize exactly what that is then it will help. > I can't see any difference but a date of installation. Really I compared > all parameters and got nothing interesting. > >>> At first glance one can say that problem is in Dell's x850 series or >>> amr(4), but we run this hardware on many other projects and they work >>> well. Also Linux on them works. >> >> OK but there is no evidence in what you posted so far that amr is >> involved in any way. There is convincing evidence that it is the mbuf >> issue. > Why are you sure this is the mbuf issue? Because that is the only problem shown in the data you posted. > For example, if there is a real > problem with amr or VM causing disk slowdown, then when it occurs the > network subsystem will have another load pattern. Instead of just quick > sending large amounts of data, the system will have to accept large > amount of sumultaneous connections waiting for data. Can this cause high > mbuf contention? I'd expect to see evidence of the main problem. >>> And few hours ago I received feed back from Andrzej Tobola, he has >>> the same problem on FreeBSD 7 with Promise ATA software mirror: >> Well, he didnt provide any evidence yet that it is the same problem, >> so let's not become confused by feelings :) > I think he is telling about 100% disk busy while processing ~5 > transfers/sec. "% busy" as reported by gstat doesn't mean what you think it does. What is the I/O response time? That's the meaningful statistic for evaluating I/O load. Also you didnt post about this. >>> So I can conclude that FreeBSD has a long standing bug in VM that >>> could be triggered when serving large amount of static data (much >>> bigger than memory size) on high rates. Possibly this only applies to >>> large files like mp3 or video. >> It is possible, we have further work to do to conclude this though. > I forgot to mention I have pmc and kgmon profiling for good and bad > times. But I have not enough knowledge to interpret it right and not > sure if it can help. pmc would be useful. > Also now I run nginx instead of lighttpd on one of the problematic > servers. It seems to work much better - sometimes there is a peaks in > disk load, but disk does not become very slow and network output does > not change. The difference of nginx is that it runs in multiple > processes, while lighttpd by default has only one process. Now I > configured lighttpd on other server to run in multiple workers. I'll see > if it helps. > > What else can i try? Still waiting on the vmstat -z output. Kris From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 08:46:33 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FB5B16A418; Wed, 17 Oct 2007 08:46:33 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id 4D5B513C461; Wed, 17 Oct 2007 08:46:31 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4715CBE6.4020604@FreeBSD.org> Date: Wed, 17 Oct 2007 10:46:30 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> In-Reply-To: <4715C5D7.7060806@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org, Alexey Popov Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 08:46:33 -0000 Kris Kennaway wrote: >> What else can i try? > > Still waiting on the vmstat -z output. Also can you please obtain vmstat -i, netstat -m and 10 seconds of representative vmstat -w output when the problem is and is not occurring? Kris From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 10:00:06 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C50216A418 for ; Wed, 17 Oct 2007 10:00:06 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82]) by mx1.freebsd.org (Postfix) with ESMTP id DAAF513C467 for ; Wed, 17 Oct 2007 10:00:04 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.14.1/8.14.1) with ESMTP id l9HA03la001942; Wed, 17 Oct 2007 20:00:03 +1000 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.14.1/8.14.1/Submit) id l9HA03vb001941; Wed, 17 Oct 2007 20:00:03 +1000 (EST) (envelope-from peter) Date: Wed, 17 Oct 2007 20:00:03 +1000 From: Peter Jeremy To: Eric Anderson Message-ID: <20071017100003.GK1191@turion.vk2pj.dyndns.org> References: <20071016113046.GA35318@eos.sc1.parodius.com> <4714A663.5010800@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="GvXjxJ+pjyke8COw" Content-Disposition: inline In-Reply-To: <4714A663.5010800@freebsd.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 10:00:06 -0000 --GvXjxJ+pjyke8COw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-Oct-16 06:54:11 -0500, Eric Anderson wrote: >will give you a good understanding of what the issue is. Essentially, your= =20 >disk is hammered making copies of all the cylinder groups, skipping those= =20 >that are 'busy', and coming back to them later. On a 200Gb disk, you could= =20 >have 1000 cylinder groups, each having to be locked, copied, unlocked, and= =20 >then checked again for any subsequent changes. The stalls you see are whe= n=20 >there are lock contentions, or disk IO issues. On a single disk (like you= r=20 >setup above), your snapshots will take forever since there is very little= =20 >random IO performance available to you. That said, there is a fair amount of scope available for improving both the creation and deletion performance. Firstly, it's not clear to me that having more than a few hundred CGs has any real benefits. There was a massive gain in moving from (effectively) a single CG in pre-FFS to a few dozen CGs in FFS as it was first introduced. Modern disks are roughly 5 orders of magnitude larger and voice-coil actuators mean that seek times are almost independent of distance. CG sizes are currently limited by the requirement that the cylinder group (including cylinder group maps) must fit into a single FS block. Removing this restriction would allow CGs to be much larger. Secondly, all the I/O during both snapshot creation and deletion is in FS-block size chunks. Increasing the I/O size would significantly increase the I/O performance. Whilst it doesn't make sense to read more than you need, there still appears to be plenty of scope to combine writes. Between these two items, I would expect potential performance gains of at least 20:1. Note that I'm not suggesting that either of these items is trivial. --=20 Peter Jeremy --GvXjxJ+pjyke8COw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQFHFd0j/opHv/APuIcRAsDTAJ98bA1hM+azlznLoQYZ5NjV9XLClQCfTFSB afHPe9YjdIXOxI2m6LbrV3o= =NI2w -----END PGP SIGNATURE----- --GvXjxJ+pjyke8COw-- From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 11:27:44 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B522616A419 for ; Wed, 17 Oct 2007 11:27:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 812AE13C455 for ; Wed, 17 Oct 2007 11:27:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l9HBRgZe062701; Wed, 17 Oct 2007 06:27:42 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <4715F1A5.50605@freebsd.org> Date: Wed, 17 Oct 2007 06:27:33 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Doug Clements References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> In-Reply-To: <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-hackers@freebsd.org Subject: Re: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:27:44 -0000 Doug Clements wrote: > Hi, > I have an new NFS server that is processing roughly 15mbit of NFS traffic > that we recently upgraded from an older 4.10 box. It has a 3-ware raid card, > and is serving NFS out a single em nic to LAN clients. The machine works > great just serving NFS, but when I try to copy data from one raid volume to > another for backups, the machine's NFS performance goes way down, and > NFSops start taking multiple seconds to perform. The file copy goes > quite > quickly, as would be expected. The console of the machine also starts to lag > pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do > the backup. > > My first impression was that I was having interrupt issues, since during the > backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60% > CPU processing interrupts). So I recompiled the kernel with polling enabled > and enabled it on the NICs. The strange thing is that polling shows enabled > in ifconfig, but systat -vm still shows the same amount of interrupts. I get > the same performance with polling enabled. > > I'm looking for some guidance on why the machine bogs so much during what > seems to me to be something that should barely impact machine performance at > all, and also why polling didn't seem to lower the number of interrupts > processed. The old machine was 6 years old running an old intel raid5, and > it handled NFS and the concurrent file copies without a sweat. > > My 3ware is setup as follows: > a 2 disk mirror, for the system > a 4 disk raid10, for /mnt/data1 > a 4 disk raid10, for /mnt/data2 > > Copyright (c) 1992-2007 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007 > admin@madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class > CPU) > Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 > Features=0xbfebfbff ,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > Features2=0x4e3bd,CX16,,,> > > AMD Features=0x20100800 > AMD Features2=0x1 > Cores per package: 2 > real memory = 4831838208 (4608 MB) > avail memory = 4125257728 (3934 MB) > ACPI APIC Table: > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) > acpi0: on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > cpu0: on acpi0 > acpi_throttle0: on cpu0 > acpi_button0: on acpi0 > acpi_button1: on acpi0 > pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 2.0 on pci0 > pci1: on pcib1 > pcib2: irq 16 at device 0.0 on pci1 > pci2: on pcib2 > pcib3: irq 16 at device 0.0 on pci2 > pci3: on pcib3 > pcib4: irq 17 at device 1.0 on pci2 > pci4: on pcib4 > pcib5: irq 18 at device 2.0 on pci2 > pci5: on pcib5 > em0: port > 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at > device 0.0 on pci5 > em0: Ethernet address: 00:15:17:21:bf:30 > em1: port > 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at > device 0.1 on pci5 > em1: Ethernet address: 00:15:17:21:bf:31 > pcib6: at device 0.3 on pci1 > pci6: on pcib6 > 3ware device driver for 9000 series storage controllers, version: > 3.60.02.012 > twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem > 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 > twa0: [GIANT-LOCKED] > twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, > Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 > pcib7: at device 3.0 on pci0 > pci7: on pcib7 > pcib8: at device 4.0 on pci0 > pci8: on pcib8 > pcib9: at device 5.0 on pci0 > pci9: on pcib9 > pcib10: at device 6.0 on pci0 > pci10: on pcib10 > pcib11: at device 7.0 on pci0 > pci11: on pcib11 > pci0: at device 8.0 (no driver attached) > pcib12: irq 16 at device 28.0 on pci0 > pci12: on pcib12 > uhci0: port 0x4080-0x409f irq 23 at device > 29.0 on pci0 > uhci0: [GIANT-LOCKED] > usb0: on uhci0 > usb0: USB revision 1.0 > uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub0: 2 ports with 2 removable, self powered > uhci1: port 0x4060-0x407f irq 22 at device > 29.1 on pci0 > uhci1: [GIANT-LOCKED] > usb1: on uhci1 > usb1: USB revision 1.0 > uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub1: 2 ports with 2 removable, self powered > uhci2: port 0x4040-0x405f irq 23 at device > 29.2 on pci0 > uhci2: [GIANT-LOCKED] > usb2: on uhci2 > usb2: USB revision 1.0 > uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub2: 2 ports with 2 removable, self powered > uhci3: port 0x4020-0x403f irq 22 at device > 29.3 on pci0 > uhci3: [GIANT-LOCKED] > usb3: on uhci3 > usb3: USB revision 1.0 > uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 > uhub3: 2 ports with 2 removable, self powered > ehci0: mem 0xf8c00400-0xf8c007ff irq 23 > at device 29.7 on pci0 > ehci0: [GIANT-LOCKED] > usb4: EHCI version 1.0 > usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 > usb4: on ehci0 > usb4: USB revision 2.0 > uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 > uhub4: 8 ports with 8 removable, self powered > pcib13: at device 30.0 on pci0 > pci13: on pcib13 > pci13: at device 12.0 (no driver attached) > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on > pci0 > ata0: on atapci0 > ata1: on atapci0 > atapci1: port > 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem > 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 > ata2: on atapci1 > ata3: on atapci1 > pci0: at device 31.3 (no driver attached) > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on > acpi0 > sio0: type 16550A > sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > sio1: type 16550A > orm0: at iomem > 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 > ppc0: cannot reserve I/O port range > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounter "TSC" frequency 2670647184 Hz quality 800 > Timecounters tick every 1.000 msec > acd0: CDRW at ata0-master UDMA33 > da0 at twa0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 100.000MB/s transfers > da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) > da1 at twa0 bus 0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 100.000MB/s transfers > da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) > da2 at twa0 bus 0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 100.000MB/s transfers > da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) > Trying to mount root from ufs:/dev/da0s1a > em1: link state changed to UP > > > [root@madonna ~]# mount > /dev/da0s1a on / (ufs, local) > devfs on /dev (devfs, local) > /dev/da0s1e on /usr (ufs, local, soft-updates) > /dev/da0s1d on /var (ufs, local, soft-updates) > /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates) > /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates) > > [root@madonna ~]# df -hi > Filesystem Size Used Avail Capacity iused ifree %iused Mounted > on > /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% / > devfs 1.0K 1.0K 0B 100% 0 0 100% /dev > /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr > /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var > /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2% > /mnt/data1 > /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7% > /mnt/data2 > > [root@madonna ~]# more /etc/exports > /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth > kelli miho deborah > /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth > kelli miho deborah > > [root@madonna ~]# ifconfig -a > em0: flags=8802 mtu 1500 > options=4b > ether 00:15:17:21:bf:30 > media: Ethernet autoselect > status: no carrier > em1: flags=8843 mtu 1500 > options=4b > inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x > ether 00:15:17:21:bf:31 > media: Ethernet 100baseTX > status: active > lo0: flags=8049 mtu 16384 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 > inet6 ::1 prefixlen 128 > inet 127.0.0.1 netmask 0xff000000 > > Anyone have any clue about what might be going on? Can you also send the output of ps -auxl? Also - do you notice this performance drop when running something like one of the network performance tools? I'd like to isolate the disk activity from the network activity for a clean test.. Eric From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 11:32:10 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6FA5C16A418 for ; Wed, 17 Oct 2007 11:32:10 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from sssup.it (ms01.sssup.it [193.205.80.99]) by mx1.freebsd.org (Postfix) with ESMTP id 8CE9613C458 for ; Wed, 17 Oct 2007 11:32:09 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from [10.30.3.4] (HELO granpasso.retis) by sssup.it (CommuniGate Pro SMTP 4.1.8) with ESMTP-TLS id 35078491; Wed, 17 Oct 2007 12:20:08 +0200 Received: from granpasso.retis (localhost.retis [127.0.0.1]) by granpasso.retis (8.14.1/8.14.1) with ESMTP id l9HA92qY098989; Wed, 17 Oct 2007 12:09:07 +0200 (CEST) (envelope-from fabio@freebsd.org) Received: (from fabio@localhost) by granpasso.retis (8.14.1/8.14.1/Submit) id l9GMNEfI074551; Wed, 17 Oct 2007 00:23:14 +0200 (CEST) (envelope-from fabio@freebsd.org) X-Authentication-Warning: granpasso.retis: fabio set sender to fabio@freebsd.org using -f Date: Wed, 17 Oct 2007 00:23:14 +0200 From: Fabio Checconi To: Karsten Behrmann Message-ID: <20071016222314.GD1243@gandalf.sssup.it> Mail-Followup-To: Karsten Behrmann , freebsd-hackers@freebsd.org References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> User-Agent: Mutt/1.4.2.3i Cc: freebsd-hackers@freebsd.org Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:32:10 -0000 > From: Karsten Behrmann > Date: Tue, Oct 16, 2007 04:10:37PM +0200 > > > Hi, > > is anybody working on the `Pluggable Disk Scheduler Project' from > > the ideas page? > I've been kicking the idea around in my head, but I'm probably newer to > everything involved than you are, so feel free to pick it up. If you want, > we can toss some ideas and code to each other, though I don't really > have anything on the latter. Thank you for your answer, I'd really like to work/discuss with you and anyone else interested in this project. > > o Where is the right place (in GEOM) for a disk scheduler? > I have spent some time at eurobsdcon talking to Kirk and phk about > this, and the result was that I now know strong proponents both for > putting it into the disk drivers and for putting it into geom ;-) > > Personally, I would put it into geom. I'll go into more detail on > this later, but basically, geom seems a better fit for "high-level" > code than a device driver, and if done properly performance penalties > should be negligible. > I'm pretty interested even in the arguments for the opposite solution; I've started from GEOM because it seemed to be a) what was proposed/ requested on the ideas page, and b) cleaner at least for a prototype. I wanted to start with some code also to evaluate the performance penalties of that approach. I am a little bit scared from the perspective of changing the queueing mechanisms that drivers use, as this kind of modifications can be difficult to write, test and maintain, but I'd really like to know what people with experience in those kernel areas think about the possibility of doing more complex io scheduling, with some sort of unified interface, at this level. As a side note, by now I've not posted any performance number because at the moment I've only access to old ata drives that would not give significative results. > > o How can anticipation be introduced into the GEOM framework? > I wouldn't focus on just anticipation, but also other types of > schedulers (I/O scheduling influenced by nice value?) > That would be interesting, especially for the background fsck case. I think that some kind of fair sharing approach should be used; as you say below a priority driven scheduler can have relations with the VFS that are difficult to track. (This problem was pointed out also in one of the follow-ups to [1].) > > o How to deal with metadata requests and other VFS issues? > Like any other disk request, though for priority-respecting > schedulers this may get rather interesting. > > [...] > > The main idea is to allow the scheduler to enqueue the requests > > having only one (other small fixed numbers can be better on some > > hardware) outstanding request and to pass new requests to its > > provider only after the service of the previous one ended. > You'll want to queue at least two requests at once. The reason for > this is performance: > Currently, drivers queue their own I/O. This means that as soon > as a request completes (on devices that don't have in-device > queues), they can fairly quickly grab a new request from their > internal queue and push it back to the device from the interrupt > handler or some other fast method. Wouldn't that require, to be sustainable (unless you want a fast dispatch every two requests,) that the driver queue is always of length two or more? In this way you ask, to refill the driver queue, the upper scheduler to dispatch a new request every time a request is taken from the driver queue and sent to the disk, or in any other moment before the request under service is completed. In this way you cannot have an anticipation mechanism, because the next request you'll want to dispatch from the upper scheduler has not yet been issued (it will be only after the one being served is completed and after the userspace application restarts.) > Having the device idle while the response percolates up the geom > stack and a new request down will likely be rather wasteful. I completely agree on that. I've only done in this way because it was the less intrusive option I could find. What can be other more efficient alternatives? (Obviously without subverting any of the existing interfaces, and allowing the anticipation of requests.) > For disks with queuing, I'd recommend trying to keep the queue > reasonably full (unless the queuing strategy says otherwise), > for disks without queuing I'd say we want to push at least one > more request down. Personally, I think the sanest design would > be to have device drivers return a "temporary I/O error" along > the lines of EAGAIN, meaning their queue is full. > This can be easily done for asynchronous requests. The troubles arise when dealing with synchronous requests... i.e., if you dispatch more than one synchronous request you are serving more than one process, and you have long seek times between the requests you have dispatched. I think we should do (I'll do as soon as I have access to some realistic test system) some measurement on real hardware, looking at what happens when we allow multiple outstanding requests. > > The example scheduler in the draft takes the following approach: > > > > o a scheduling GEOM class is introduced. It can be stacked on > > top of disk geoms, and schedules all the requests coming > > from its consumers. I'm not absolutely sure that a new class > > is really needed but I think that it can simplify testing and > > experimenting with various solutions on the scheduler placement. > Probably, though we'll want to make sure that they stack on top of > (or are inside of?) the geoms talking to the disks, because it rarely > makes sense to put a queuing geom on top of, say, a disklabel geom. > > The advantage of making it a full geom is configurability. You would > be able to swap out a scheduler at runtime, select different sched- > ulers for different disks, and potentially even test new schedulers > without rebooting (though you wouldn't want to do that for benchmarks) > Ok. > > o Requests coming from consumers are passed down immediately > > if there is no other request under service, otherwise they > > are queued in a bioq. > This is specific to the anticipatory scheduler. I would say in more > general terms: > - A queuing geom is to push all requests that it wants serviced down > towards the disk, until the disk reports a queue full. A queuing geom > is allowed to hold back requests even when the driver queue is not > full yet, if it does not want the disk to attempt such I/O yet (such > as the anticipatory scheduler waiting for another disk request near > the last one, or the process-priority scheduler holding back a low- > priority request that would potentially cause a long seek, until io > has been idle) > This dispels phk's anti-geom argument of "it will be inefficient > because it will take longer for a new request to get to the driver" - > if the queuing strategy had wanted the request to be sent to the > drive, it would already have sent it. (The exception is that the disk > will have its internal queue a little more empty than it could be - > not a big issue with queue sizes of 8 or 16) > Well, it will take longer for new requests to get to the driver, because we are inserting a whole new geom (or a new layer into an existing one) in the path of a request. Of course the benefits should be in the time it takes to complete the request (or in other properties of the service, such as fairness or something like bounded service times.) The mechanism you describe of an high level scheduler that fills an underlying queue is somehow similar to what Hybrid did, with its two-stage pipelined structure, but to the best of my knowledge it did not work with synchronous requests, for the same problems described above. > [...] > > So, as I've said, I'd like to know what you think about the subject, > > if I'm missing something, if there is some kind of interest on this > > and if/how can this work proceed. > I would think that this would be quite useful, but I don't think my > voice counts for much ;-) > It would help > - servers where anticipatory performs better than elevator > - realtime environments that need a scheduler fitting their needs > - the background fsck, if someone implements a "priority" scheduler > > Anyway, that's a quick dump of my thoughts on the subject so far, > I've myself wanted to get started on this but didn't get around to > it yet (I'm fairly new to FreeBSD). > If you want to hash some ideas out with me, I'll be watching my inbox, > this ML, and you can reach me on IRC as BearPerson on freenode, > quakenet, undernet, or whatever else you ask me to connect to, or via > whatever other method convenient to you ;) > Thank you very much, your help is very much appreciated. I'll try to set up some kind of testing environment and share some result. [1] http://lists.freebsd.org/pipermail/freebsd-geom/2007-January/001854.html From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 11:33:41 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E50316A417 for ; Wed, 17 Oct 2007 11:33:41 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 07CC213C447 for ; Wed, 17 Oct 2007 11:33:40 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l9HBXdjF071289; Wed, 17 Oct 2007 06:33:40 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <4715F30A.5080102@freebsd.org> Date: Wed, 17 Oct 2007 06:33:30 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Kostik Belousov References: <20071016113046.GA35318@eos.sc1.parodius.com> <4714A663.5010800@freebsd.org> <20071017100003.GK1191@turion.vk2pj.dyndns.org> <20071017101400.GH6511@deviant.kiev.zoral.com.ua> In-Reply-To: <20071017101400.GH6511@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:33:41 -0000 Kostik Belousov wrote: > On Wed, Oct 17, 2007 at 08:00:03PM +1000, Peter Jeremy wrote: >> On 2007-Oct-16 06:54:11 -0500, Eric Anderson wrote: >>> will give you a good understanding of what the issue is. Essentially, your >>> disk is hammered making copies of all the cylinder groups, skipping those >>> that are 'busy', and coming back to them later. On a 200Gb disk, you could >>> have 1000 cylinder groups, each having to be locked, copied, unlocked, and >>> then checked again for any subsequent changes. The stalls you see are when >>> there are lock contentions, or disk IO issues. On a single disk (like your >>> setup above), your snapshots will take forever since there is very little >>> random IO performance available to you. >> That said, there is a fair amount of scope available for improving >> both the creation and deletion performance. >> >> Firstly, it's not clear to me that having more than a few hundred CGs >> has any real benefits. There was a massive gain in moving from >> (effectively) a single CG in pre-FFS to a few dozen CGs in FFS as it >> was first introduced. Modern disks are roughly 5 orders of magnitude >> larger and voice-coil actuators mean that seek times are almost >> independent of distance. CG sizes are currently limited by the >> requirement that the cylinder group (including cylinder group maps) >> must fit into a single FS block. Removing this restriction would >> allow CGs to be much larger. >> >> Secondly, all the I/O during both snapshot creation and deletion is >> in FS-block size chunks. Increasing the I/O size would significantly >> increase the I/O performance. Whilst it doesn't make sense to read >> more than you need, there still appears to be plenty of scope to >> combine writes. >> >> Between these two items, I would expect potential performance gains >> of at least 20:1. >> >> Note that I'm not suggesting that either of these items is trivial. > This is, unfortunately, quite true. Allowing non-atomic updates of the > cg block means a lot of complications in the softupdate code, IMHO. I agree with all the above. I think it has not been done because of exactly what Kostik says. I really think that the CG max size is *way* too small now, and should be about 10-50 times larger, but performance tests would need to be run. Eric From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 11:37:30 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6AAC516A503 for ; Wed, 17 Oct 2007 11:37:30 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from signal.itea.ntnu.no (signal.itea.ntnu.no [129.241.190.231]) by mx1.freebsd.org (Postfix) with ESMTP id E92CF13C448 for ; Wed, 17 Oct 2007 11:37:29 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from localhost (localhost [127.0.0.1]) by signal.itea.ntnu.no (Postfix) with ESMTP id 901DE33703; Wed, 17 Oct 2007 13:07:08 +0200 (CEST) Received: from caracal.stud.ntnu.no (caracal.stud.ntnu.no [129.241.56.185]) by signal.itea.ntnu.no (Postfix) with ESMTP; Wed, 17 Oct 2007 13:07:08 +0200 (CEST) Received: by caracal.stud.ntnu.no (Postfix, from userid 2312) id C2C366240FD; Wed, 17 Oct 2007 13:07:15 +0200 (CEST) Date: Wed, 17 Oct 2007 13:07:15 +0200 From: Ulf Lilleengen To: Karsten Behrmann Message-ID: <20071017110715.GA25075@stud.ntnu.no> References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> User-Agent: Mutt/1.5.9i X-Content-Scanned: with sophos and spamassassin at mailgw.ntnu.no. Cc: freebsd-hackers@freebsd.org, Fabio Checconi Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 11:37:30 -0000 On tir, okt 16, 2007 at 04:10:37 +0200, Karsten Behrmann wrote: > > Hi, > > is anybody working on the `Pluggable Disk Scheduler Project' from > > the ideas page? > I've been kicking the idea around in my head, but I'm probably newer to > everything involved than you are, so feel free to pick it up. If you want, > we can toss some ideas and code to each other, though I don't really > have anything on the latter. > > [...] > > After reading [1], [2] and its follow-ups the main problems that > > need to be addressed seem to be: > > > > o is working on disk scheduling worth at all? > Probably, one of the main applications would be to make the background > fsck a little more well-behaved. I agree, as I said before, the ability to give I/O priorities is probably one of the most important things. > > > o Where is the right place (in GEOM) for a disk scheduler? [...] > > > o How can anticipation be introduced into the GEOM framework? > I wouldn't focus on just anticipation, but also other types of > schedulers (I/O scheduling influenced by nice value?) > > > o What can be an interface for disk schedulers? > good question, but geom seems a good start ;) > > > o How to deal with devices that handle multiple request per time? > Bad news first: this is most disks out there, in a way ;) > SCSI has tagged queuing, ATA has native command queing or > whatever the ata people came up over their morning coffee today. > I'll mention a bit more about this further down. > > > o How to deal with metadata requests and other VFS issues? > Like any other disk request, though for priority-respecting > schedulers this may get rather interesting. > > [...] > > The main idea is to allow the scheduler to enqueue the requests > > having only one (other small fixed numbers can be better on some > > hardware) outstanding request and to pass new requests to its > > provider only after the service of the previous one ended. [...] > - servers where anticipatory performs better than elevator > - realtime environments that need a scheduler fitting their needs > - the background fsck, if someone implements a "priority" scheduler Apache is actally a good candidate according to the old antipacitory design document ( not sure of it's relevance today, but...) Over to a more general view of it's architecture: When I looked at this project for the first time, I was under the impression that this would be best done in a GEOM class. However, I think the approach that was taken in the Hybrid project is better because of the following reasons: - It makes it possible to use by _both_ GEOM classes and device drivers (Which might use some other scheduler type?). - Does not remove any configuratbility, since changing etc. can be done by user with sysctl. - Could make it possible for a GEOM class to decide for itself which scheduler it wants to use (most GEOM classes uses the standard bioq_disksort interface in disk_subr.c). - The ability to stack a GEOM class with a scheduler could easily be "emulated" by creating a GEOM class to utilize the disksched framework. All in all, I just think this approach gives more flexibility than putting it in a GEOM class that have to be added manually by a user. Just my thought on this. Also, I got my test-box up again today, and will be trying your patch as soon as I've upgraded it to CURRENT Fabio. -- Ulf Lilleengen From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 08:07:46 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A480516A419; Wed, 17 Oct 2007 08:07:46 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from hermes.hw.ru (hermes.hw.ru [80.68.240.91]) by mx1.freebsd.org (Postfix) with ESMTP id 790C513C447; Wed, 17 Oct 2007 08:07:44 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from [80.68.244.40] (account a_popov@rbc.ru [80.68.244.40] verified) by hermes.hw.ru (CommuniGate Pro SMTP 5.0.13) with ESMTPA id 194588927; Wed, 17 Oct 2007 12:07:50 +0400 Message-ID: <4715C297.1020905@chistydom.ru> Date: Wed, 17 Oct 2007 12:06:47 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> In-Reply-To: <4715035D.2090802@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 17 Oct 2007 11:50:21 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 08:07:46 -0000 Hi. Kris Kennaway wrote: >>>>>> After some time of running under high load disk performance become >>>>>> expremely poor. At that periods 'systat -vm 1' shows something like >>>>>> this: >>>> This web service is similiar to YouTube. This server is video store. I >>>> have around 200G of *.flv (flash video) files on the server >>>> I run lighttpd as a web server. Disk load is usually around 50%, >>>> network >>>> output 100Mbit/s, 100 simultaneous connections. CPU is mostly idle. >> This is very unlikely, because I have 5 another video storage servers >> of the same hardware and software configurations and they feel good. > Clearly something is different about them, though. If you can > characterize exactly what that is then it will help. I can't see any difference but a date of installation. Really I compared all parameters and got nothing interesting. >> At first glance one can say that problem is in Dell's x850 series or >> amr(4), but we run this hardware on many other projects and they work >> well. Also Linux on them works. > > OK but there is no evidence in what you posted so far that amr is > involved in any way. There is convincing evidence that it is the mbuf > issue. Why are you sure this is the mbuf issue? For example, if there is a real problem with amr or VM causing disk slowdown, then when it occurs the network subsystem will have another load pattern. Instead of just quick sending large amounts of data, the system will have to accept large amount of sumultaneous connections waiting for data. Can this cause high mbuf contention? > >> And few hours ago I received feed back from Andrzej Tobola, he has the >> same problem on FreeBSD 7 with Promise ATA software mirror: > Well, he didnt provide any evidence yet that it is the same problem, so > let's not become confused by feelings :) I think he is telling about 100% disk busy while processing ~5 transfers/sec. >> So I can conclude that FreeBSD has a long standing bug in VM that >> could be triggered when serving large amount of static data (much >> bigger than memory size) on high rates. Possibly this only applies to >> large files like mp3 or video. > It is possible, we have further work to do to conclude this though. I forgot to mention I have pmc and kgmon profiling for good and bad times. But I have not enough knowledge to interpret it right and not sure if it can help. Also now I run nginx instead of lighttpd on one of the problematic servers. It seems to work much better - sometimes there is a peaks in disk load, but disk does not become very slow and network output does not change. The difference of nginx is that it runs in multiple processes, while lighttpd by default has only one process. Now I configured lighttpd on other server to run in multiple workers. I'll see if it helps. What else can i try? With best regards, Alexey Popov From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 12:19:18 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E040716A418 for ; Wed, 17 Oct 2007 12:19:18 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from sssup.it (ms01.sssup.it [193.205.80.99]) by mx1.freebsd.org (Postfix) with ESMTP id 402DF13C465 for ; Wed, 17 Oct 2007 12:19:17 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from [10.30.3.4] (HELO granpasso.retis) by sssup.it (CommuniGate Pro SMTP 4.1.8) with ESMTP-TLS id 35081401; Wed, 17 Oct 2007 14:07:44 +0200 Received: from granpasso.retis (localhost.retis [127.0.0.1]) by granpasso.retis (8.14.1/8.14.1) with ESMTP id l9HCJ8h6000500; Wed, 17 Oct 2007 14:19:08 +0200 (CEST) (envelope-from fabio@freebsd.org) Received: (from fabio@localhost) by granpasso.retis (8.14.1/8.14.1/Submit) id l9HCJ7TM000499; Wed, 17 Oct 2007 14:19:07 +0200 (CEST) (envelope-from fabio@freebsd.org) X-Authentication-Warning: granpasso.retis: fabio set sender to fabio@freebsd.org using -f Date: Wed, 17 Oct 2007 14:19:07 +0200 From: Fabio Checconi To: Ulf Lilleengen Message-ID: <20071017121907.GL99087@gandalf.sssup.it> Mail-Followup-To: Ulf Lilleengen , Karsten Behrmann , freebsd-hackers@freebsd.org References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> <20071017110715.GA25075@stud.ntnu.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071017110715.GA25075@stud.ntnu.no> User-Agent: Mutt/1.4.2.3i Cc: Karsten Behrmann , freebsd-hackers@freebsd.org Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 12:19:19 -0000 > From: Ulf Lilleengen > Date: Wed, Oct 17, 2007 01:07:15PM +0200 > > On tir, okt 16, 2007 at 04:10:37 +0200, Karsten Behrmann wrote: > Over to a more general view of it's architecture: > > When I looked at this project for the first time, I was under the impression > that this would be best done in a GEOM class. > > However, I think the approach that was taken in the Hybrid project is better Ok. I think that such a solution requires a lot more effort on the design and coding sides, as it requires the modification of the drivers and can bring us problems with locking and with the queueing assumptions that may vary on a per-driver basis. Maybe I've not enough experience/knowledge of the driver subsystem, but I would not remove the queueing that is done now by the drivers (think of ata freezepoints,) but instead I'd like to try to grab the requests before they get to the driver (e.g., in/before their d_strategy call) and have some sort of pull mechanism when requests complete (still don't have any (serious) idea on that, I fear that the right place to do that, for locking issues and so on, can be driver dependent.) Any ideas on that? Which drivers can be good starting points to try to write down some code? > Also, I got my test-box up again today, and will be trying your patch as soon > as I've upgraded it to CURRENT Fabio. Thank you very much! Please consider that my primary concern with the patch was its interface, the algorithm is just an example (it should give an idea of the performance loss due to the mechanism overhead with async requests, and some improvement on greedy sync loads.) From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 12:43:30 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7207816A418 for ; Wed, 17 Oct 2007 12:43:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from gnome.kiev.sovam.com (gnome.kiev.sovam.com [212.109.32.24]) by mx1.freebsd.org (Postfix) with ESMTP id 168D513C480 for ; Wed, 17 Oct 2007 12:43:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com ([62.64.120.197]) by gnome.kiev.sovam.com with esmtp (Exim 4.67 (FreeBSD)) (envelope-from ) id 1Ii5ul-00099W-P0; Wed, 17 Oct 2007 13:14:11 +0300 Received: from [212.82.216.226] (helo=deviant.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1Ii5uj-000Ii5-UX; Wed, 17 Oct 2007 13:14:11 +0300 Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id l9HAE0WS024066; Wed, 17 Oct 2007 13:14:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.1/8.14.1/Submit) id l9HAE07V024065; Wed, 17 Oct 2007 13:14:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 17 Oct 2007 13:14:00 +0300 From: Kostik Belousov To: Peter Jeremy Message-ID: <20071017101400.GH6511@deviant.kiev.zoral.com.ua> References: <20071016113046.GA35318@eos.sc1.parodius.com> <4714A663.5010800@freebsd.org> <20071017100003.GK1191@turion.vk2pj.dyndns.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uAgJxtfIS94j9H4T" Content-Disposition: inline In-Reply-To: <20071017100003.GK1191@turion.vk2pj.dyndns.org> User-Agent: Mutt/1.4.2.3i X-Scanner-Signature: fa1e9f04260dd27375e84b1224048298 X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 1629 [Oct 16 2007] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-hackers@freebsd.org Subject: Re: Filesystem snapshots dog slow X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 12:43:30 -0000 --uAgJxtfIS94j9H4T Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 17, 2007 at 08:00:03PM +1000, Peter Jeremy wrote: > On 2007-Oct-16 06:54:11 -0500, Eric Anderson wrote: > >will give you a good understanding of what the issue is. Essentially, yo= ur=20 > >disk is hammered making copies of all the cylinder groups, skipping thos= e=20 > >that are 'busy', and coming back to them later. On a 200Gb disk, you cou= ld=20 > >have 1000 cylinder groups, each having to be locked, copied, unlocked, a= nd=20 > >then checked again for any subsequent changes. The stalls you see are w= hen=20 > >there are lock contentions, or disk IO issues. On a single disk (like y= our=20 > >setup above), your snapshots will take forever since there is very littl= e=20 > >random IO performance available to you. >=20 > That said, there is a fair amount of scope available for improving > both the creation and deletion performance. >=20 > Firstly, it's not clear to me that having more than a few hundred CGs > has any real benefits. There was a massive gain in moving from > (effectively) a single CG in pre-FFS to a few dozen CGs in FFS as it > was first introduced. Modern disks are roughly 5 orders of magnitude > larger and voice-coil actuators mean that seek times are almost > independent of distance. CG sizes are currently limited by the > requirement that the cylinder group (including cylinder group maps) > must fit into a single FS block. Removing this restriction would > allow CGs to be much larger. >=20 > Secondly, all the I/O during both snapshot creation and deletion is > in FS-block size chunks. Increasing the I/O size would significantly > increase the I/O performance. Whilst it doesn't make sense to read > more than you need, there still appears to be plenty of scope to > combine writes. >=20 > Between these two items, I would expect potential performance gains > of at least 20:1. >=20 > Note that I'm not suggesting that either of these items is trivial. This is, unfortunately, quite true. Allowing non-atomic updates of the cg block means a lot of complications in the softupdate code, IMHO. --uAgJxtfIS94j9H4T Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHFeBnC3+MBN1Mb4gRAtSOAKDuOFuqcDEHtsPa2r4oYii2aZeYgwCgoYZC fMAT7d/+eHzEZSeLe72yyzM= =zclz -----END PGP SIGNATURE----- --uAgJxtfIS94j9H4T-- From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 13:09:29 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D726916A419; Wed, 17 Oct 2007 13:09:29 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from merke.itea.ntnu.no (merke.itea.ntnu.no [129.241.7.61]) by mx1.freebsd.org (Postfix) with ESMTP id 580E113C44B; Wed, 17 Oct 2007 13:09:29 +0000 (UTC) (envelope-from lulf@stud.ntnu.no) Received: from localhost (localhost [127.0.0.1]) by merke.itea.ntnu.no (Postfix) with ESMTP id E420213CFBD; Wed, 17 Oct 2007 15:09:27 +0200 (CEST) Received: from caracal.stud.ntnu.no (caracal.stud.ntnu.no [129.241.56.185]) by merke.itea.ntnu.no (Postfix) with ESMTP; Wed, 17 Oct 2007 15:09:27 +0200 (CEST) Received: by caracal.stud.ntnu.no (Postfix, from userid 2312) id 227A96240FD; Wed, 17 Oct 2007 15:09:35 +0200 (CEST) Date: Wed, 17 Oct 2007 15:09:35 +0200 From: Ulf Lilleengen To: Fabio Checconi Message-ID: <20071017130934.GA26180@stud.ntnu.no> References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> <20071017110715.GA25075@stud.ntnu.no> <20071017121907.GL99087@gandalf.sssup.it> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071017121907.GL99087@gandalf.sssup.it> User-Agent: Mutt/1.5.9i X-Content-Scanned: with sophos and spamassassin at mailgw.ntnu.no. Cc: freebsd-hackers@freebsd.org, s223560@studenti.ing.unipi.it, luigi@FreeBSD.org Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 13:09:29 -0000 On ons, okt 17, 2007 at 02:19:07 +0200, Fabio Checconi wrote: > > From: Ulf Lilleengen > > Date: Wed, Oct 17, 2007 01:07:15PM +0200 > > > > On tir, okt 16, 2007 at 04:10:37 +0200, Karsten Behrmann wrote: > > Over to a more general view of it's architecture: > > > > When I looked at this project for the first time, I was under the impression > > that this would be best done in a GEOM class. > > > > However, I think the approach that was taken in the Hybrid project is better > > Ok. I think that such a solution requires a lot more effort on the > design and coding sides, as it requires the modification of the > drivers and can bring us problems with locking and with the queueing > assumptions that may vary on a per-driver basis. > I completely agree with the issue of converting device drivers, but at least it will be an _optional_ possibility (Having different scheduler plugins could make this possible). One does not necessary need to convert the drivers. > Maybe I've not enough experience/knowledge of the driver subsystem, > but I would not remove the queueing that is done now by the drivers > (think of ata freezepoints,) but instead I'd like to try to grab > the requests before they get to the driver (e.g., in/before their > d_strategy call) and have some sort of pull mechanism when requests > complete (still don't have any (serious) idea on that, I fear that > the right place to do that, for locking issues and so on, can be > driver dependent.) Any ideas on that? Which drivers can be good > starting points to try to write down some code? > If you look at it, Hybrid is just a generalization of the existing bioq_* API already defined. And this API is used by GEOM classes _before_ device drivers get the requests AFAIK. For a simple example on a driver, the md-driver might be a good place to look. Note that I have little experience and knowledge of the driver subsystem myself. Also note (from the Hybrid page): * we could not provide support for non work-conserving schedulers, due to a couple of reasons: 1. the assumption, in some drivers, that bioq_disksort() will make requests immediately available (so a subsequent bioq_first() will not return NULL). 2. the fact that there is no bioq_lock()/bioq_unlock(), so the scheduler does not have a safe way to generate requests for a given queue. This certainly argues for having this in the GEOM layer, but perhaps it's possible to change the assumtions done in some drivers? The locking issue should perhaps be better planned though, and an audit of the driver disksort code is necessary. Also: * as said, the ATA driver in 6.x/7.x moves the disksort one layer below the one we are working at, so this particular work won't help on ATA-based 6.x machines. We should figure out how to address this, because the work done at that layer is mostly a replica of the bioq_*() API. So, I see this can get a bit messy thinking of that the ATA drivers does disksorts on its own, but perhaps it would be possible to fix this by letting changing the general ATA driver to use it's own pluggable scheduler. Anyway, I shouldn't demand that you do this, especially since I don't have any code or anything to show to, and because you decide what you want to do. However, I'd hate to see the Hybrid effort go to waste :) I was hoping some of the authors of the project would reply with their thoughts, so I CC'ed them. -- Ulf Lilleengen From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 13:16:47 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 132F516A419 for ; Wed, 17 Oct 2007 13:16:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 01A2D13C457 for ; Wed, 17 Oct 2007 13:16:46 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id 9659C1A4D87; Wed, 17 Oct 2007 06:16:46 -0700 (PDT) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Wed, 17 Oct 2007 09:16:18 -0400 User-Agent: KMail/1.9.7 References: <20071016094118.GE5411@hoeg.nl> In-Reply-To: <20071016094118.GE5411@hoeg.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200710170916.18788.jhb@freebsd.org> Cc: Ed Schouten Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 13:16:47 -0000 On Tuesday 16 October 2007 05:41:18 am Ed Schouten wrote: > Hello, > > I asked the following question on questions@, but as requested, I'll > forward this question to this list, because of its technical nature. > > ----- Forwarded message from Ed Schouten ----- > > Date: Mon, 15 Oct 2007 23:13:01 +0200 > > From: Ed Schouten > > To: freebsd-questions@freebsd.org > > Subject: Inner workings of turnstiles and sleepqueues > > > > Hello, > > > > For some reason, I want to understand how the queueing of blocked > > threads in the kernel works when waiting for a lock, which is if I > > understand correctly done by the turnstiles and sleepqueues. I'm the > > proud owner of The Design and Implementation of the FreeBSD Operating > > System book, but for some reason, I can't find anything about it in the > > book. > > > > Is there a way to obtain information about how they work? I already read > > the source somewhat, but that shouldn't be an ideal solution, in my > > opinion. The best option right now is to read the code. There are some comments in both the headers and implementation. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 13:49:12 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 611B716A41A; Wed, 17 Oct 2007 13:49:12 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.freebsd.org (Postfix) with ESMTP id 3424913C47E; Wed, 17 Oct 2007 13:49:11 +0000 (UTC) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.13.6) with ESMTP id l9HDEfiJ061102; Wed, 17 Oct 2007 06:14:41 -0700 (PDT) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id l9HDEfKQ061101; Wed, 17 Oct 2007 06:14:41 -0700 (PDT) (envelope-from rizzo) Date: Wed, 17 Oct 2007 06:14:41 -0700 From: Luigi Rizzo To: Ulf Lilleengen Message-ID: <20071017061441.B60901@xorpc.icir.org> References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> <20071017110715.GA25075@stud.ntnu.no> <20071017121907.GL99087@gandalf.sssup.it> <20071017130934.GA26180@stud.ntnu.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20071017130934.GA26180@stud.ntnu.no>; from lulf@stud.ntnu.no on Wed, Oct 17, 2007 at 03:09:35PM +0200 Cc: freebsd-hackers@freebsd.org, s223560@studenti.ing.unipi.it, luigi@freebsd.org, Fabio Checconi Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 13:49:12 -0000 On Wed, Oct 17, 2007 at 03:09:35PM +0200, Ulf Lilleengen wrote: ... discussion on Hybrid vs. GEOM as a suitable location for ... pluggable disk schedulers > However, I'd hate to see the Hybrid effort go to waste :) I was hoping some > of the authors of the project would reply with their thoughts, so I CC'ed > them. we are in good contact with Fabio and i am monitoring the discussion, don't worry. cheers luigi From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 14:40:15 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0C6E16A46B for ; Wed, 17 Oct 2007 14:40:15 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from sssup.it (ms01.sssup.it [193.205.80.99]) by mx1.freebsd.org (Postfix) with ESMTP id 6F93A13C478 for ; Wed, 17 Oct 2007 14:40:13 +0000 (UTC) (envelope-from fabio@freebsd.org) Received: from [10.30.3.4] (HELO granpasso.retis) by sssup.it (CommuniGate Pro SMTP 4.1.8) with ESMTP-TLS id 35086095; Wed, 17 Oct 2007 16:28:31 +0200 Received: from granpasso.retis (localhost.retis [127.0.0.1]) by granpasso.retis (8.14.1/8.14.1) with ESMTP id l9HEdtjH085758; Wed, 17 Oct 2007 16:39:55 +0200 (CEST) (envelope-from fabio@freebsd.org) Received: (from fabio@localhost) by granpasso.retis (8.14.1/8.14.1/Submit) id l9HEdst2085757; Wed, 17 Oct 2007 16:39:54 +0200 (CEST) (envelope-from fabio@freebsd.org) X-Authentication-Warning: granpasso.retis: fabio set sender to fabio@freebsd.org using -f Date: Wed, 17 Oct 2007 16:39:54 +0200 From: Fabio Checconi To: Ulf Lilleengen Message-ID: <20071017143954.GU99087@gandalf.sssup.it> Mail-Followup-To: Ulf Lilleengen , freebsd-hackers@freebsd.org, s223560@studenti.ing.unipi.it, luigi@FreeBSD.org References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> <20071017110715.GA25075@stud.ntnu.no> <20071017121907.GL99087@gandalf.sssup.it> <20071017130934.GA26180@stud.ntnu.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071017130934.GA26180@stud.ntnu.no> User-Agent: Mutt/1.4.2.3i Cc: freebsd-hackers@freebsd.org, s223560@studenti.ing.unipi.it, luigi@freebsd.org Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 14:40:15 -0000 > From: Ulf Lilleengen > Date: Wed, Oct 17, 2007 03:09:35PM +0200 > > On ons, okt 17, 2007 at 02:19:07 +0200, Fabio Checconi wrote: > > Maybe I've not enough experience/knowledge of the driver subsystem, [...] > If you look at it, Hybrid is just a generalization of the existing > bioq_* API already defined. And this API is used by GEOM classes _before_ > device drivers get the requests AFAIK. > I looked at the Hybrid code, but I don't think that the bioq_* family of calls can be the right place to start, for the problems experienced during the Hybrid development with locking/anticipation and because you can have the same request passing through multiple bioqs during its path to the device (e.g., two stacked geoms using two different bioqs and then a device driver using bioq_* to organize its queue, or geoms using more than one bioq, like raid3; I think the complexity can become unmanageable.) One could even think to configure each single bioq in the system, but things can get very complex in this way. > For a simple example on a driver, the md-driver might be a good place to > look. Note that I have little experience and knowledge of the driver > subsystem myself. > I'll take a look, thanks. > Also note (from the Hybrid page): > * we could not provide support for non work-conserving schedulers, due to a [...] > > This certainly argues for having this in the GEOM layer, but perhaps it's > possible to change the assumtions done in some drivers? The locking issue > should perhaps be better planned though, and an audit of the driver disksort > code is necessary. > I need some more time to think about that :) > Also: > * as said, the ATA driver in 6.x/7.x moves the disksort one layer below the > one we are working at, so this particular work won't help on ATA-based 6.x > machines. > We should figure out how to address this, because the work done at that > layer is mostly a replica of the bioq_*() API. > > So, I see this can get a bit messy thinking of that the ATA drivers does > disksorts on its own, but perhaps it would be possible to fix this by letting > changing the general ATA driver to use it's own pluggable scheduler. > > Anyway, I shouldn't demand that you do this, especially since I don't have > any code or anything to show to, and because you decide what you want to do. I still cannot say if a GEOM scheduler is better than a scheduler put at a lower level, or if the bioq_* interface is better than any other alternative, so your suggestions are welcome. Moreover I'd really like to discuss/work together, or at least do things with some agreement on them. If I'll have the time to experiment with more than one solution I'll be happy to do that. > However, I'd hate to see the Hybrid effort go to waste :) I was hoping some > of the authors of the project would reply with their thoughts, so I CC'ed > them. Well, the work done on Hybrid had also interesting aspects from the algorithm side... but that's another story... From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 16:31:38 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 968DA16A421 for ; Wed, 17 Oct 2007 16:31:38 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id 4B07F13C48D for ; Wed, 17 Oct 2007 16:31:38 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by an-out-0708.google.com with SMTP id c24so225735ana for ; Wed, 17 Oct 2007 09:31:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=dSuwU9oPDJmUcmbzyl4NjqaduYgzMDu0M3xrzpjWfWk=; b=bW/VD0UHi5OVl2FuG/6M4oSRV7cpTUFk0BxN6khUt2VJvB6BCkNNGeY3a9ls0cv3ijU7+gTLGqHlKjels4ejGlB1ZjsJOrf52cTPGP6FkhRAaO3uuBUEeIIQRsEAC56+INkwHMBvFzkx7v/CRN6SEWrpEjy+1F4UqETrrhd9cR0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=hxZd+bEGjPb6R25nQOo4snc5eXMpEbelLUTr2yu4UVAYkvp/kViDQqpxeTTbXA4Efj4bPgN3/XVOBfOpruBzvCwoiOdzIb29Gxk6ScBvfkieqRClmlPYcx42GdDmmHHHixFGK15wx7NSfxu9OMCRHpRQFP5i6oOPaadL/YbInew= Received: by 10.142.148.7 with SMTP id v7mr2615790wfd.1192638695927; Wed, 17 Oct 2007 09:31:35 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Wed, 17 Oct 2007 09:31:35 -0700 (PDT) Message-ID: <54da514e0710170931r1b558bedi69e1bba50807f465@mail.gmail.com> Date: Wed, 17 Oct 2007 09:31:35 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org In-Reply-To: <4715F1A5.50605@freebsd.org> MIME-Version: 1.0 References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> <4715F1A5.50605@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 16:31:38 -0000 > > Can you also send the output of ps -auxl? > > Also - do you notice this performance drop when running something like > one of the network performance tools? I'd like to isolate the disk > activity from the network activity for a clean test.. > I tested this with iperf, and while I did see some nfs performance degradation, it did not bog the machine or delay the terminal in the same way. NFS requests were still processed in an acceptable fashion. It was still responsive to commands and ran processes just fine. I would have to say it performed as to be expected during iperf tests (on which I got about 85mbit/sec, which is also to be expected). Interrupts went down to about 2000/sec on em, but the machine did not hang. Something I've noticed is that running 'systat -vm' seems to be part of the problem. If I run the file copy by itself with rdist, it's fast and runs ok. If I run it with systat -vm going, this is when the interrupts jump way up and the machine starts to delay badly. Noticing this, I tried running 'sysctl -a' during the file copy, thinking there was some problem with polling the kernel for certain statistics. Sure enough, sysctl -a delays at 2 spots. Once right after "kern.random.sys.harvest.swi: 0" and once again after "debug.hashstat.nchash: 131072 61777 6 4713". While it is delayed here (for a couple seconds for each one) the machine is totally hung. Maybe this is a statistics polling issue? Maybe the machine is delayed just long enough in systat -vm to make the nfs clients retry, causing a storm of interrupts? Other systat modes do not seem to cause the same problem (pigs, icmp, ifstat). I do not think the ps or systat output is very accurate, since I can't get them to run when the machine is hung up. I type in the command, but it does not run until the machine springs back to life. I'm not sure how this will affect measurements. http://toric.loungenet.org/~doug/sysctl-a http://toric.loungenet.org/~doug/psauxl http://toric.loungenet.org/~doug/systat-vm My real confusion lies in why there are still em interrupts at all, with polling on. Thanks! --Doug From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 19:53:55 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E773016A420; Wed, 17 Oct 2007 19:53:55 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from comtv.ru (comtv.ru [217.10.32.17]) by mx1.freebsd.org (Postfix) with ESMTP id A49AA13C481; Wed, 17 Oct 2007 19:53:54 +0000 (UTC) (envelope-from lol@chistydom.ru) X-UCL: actv Received: from yoda.org.ru ([83.167.98.162] verified) by comtv.ru (CommuniGate Pro SMTP 4.1.8) with ESMTP id 6506471; Wed, 17 Oct 2007 22:53:52 +0400 Received: from [192.168.102.10] (unknown [192.168.102.10]) (Authenticated sender: llp@soekris.ru) by yoda.org.ru (Postfix) with ESMTP id 6622228BD6; Wed, 17 Oct 2007 22:54:00 +0400 (MSD) Message-ID: <47165A01.1030806@chistydom.ru> Date: Wed, 17 Oct 2007 22:52:49 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Kris Kennaway References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> In-Reply-To: <4715C5D7.7060806@FreeBSD.org> Content-Type: multipart/mixed; boundary="------------040405020607060803010501" X-Mailman-Approved-At: Wed, 17 Oct 2007 21:12:56 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 19:53:56 -0000 This is a multi-part message in MIME format. --------------040405020607060803010501 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Kris Kennaway wrote: >>>> And few hours ago I received feed back from Andrzej Tobola, he has >>>> the same problem on FreeBSD 7 with Promise ATA software mirror: >>> Well, he didnt provide any evidence yet that it is the same problem, >>> so let's not become confused by feelings :) >> I think he is telling about 100% disk busy while processing ~5 >> transfers/sec. > > "% busy" as reported by gstat doesn't mean what you think it does. What > is the I/O response time? That's the meaningful statistic for > evaluating I/O load. Also you didnt post about this. At the problematic time the disk felt to be very slow, processes all were in reading disk state and vmstat proved it by the % numbers. >>>> So I can conclude that FreeBSD has a long standing bug in VM that >>>> could be triggered when serving large amount of static data (much >>>> bigger than memory size) on high rates. Possibly this only applies >>>> to large files like mp3 or video. >>> It is possible, we have further work to do to conclude this though. >> I forgot to mention I have pmc and kgmon profiling for good and bad >> times. But I have not enough knowledge to interpret it right and not >> sure if it can help. > pmc would be useful. Unfortunately i've lost pmc profiling results. I'll try to collect it again later. See vmstats in attach (vmstat -z; netstat -m; vmstat -i; vmstat -w 1 | head -11;). Also you can see kgmon profiling results at: http://83.167.98.162/gprof/ With best regards, Alexey Popov --------------040405020607060803010501 Content-Type: text/plain; name="vmstat-bad.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vmstat-bad.txt" ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 240, 0, 71, 4, 71, 0 UMA Zones: 376, 0, 71, 9, 71, 0 UMA Slabs: 128, 0, 1011, 62, 243081, 0 UMA RCntSlabs: 128, 0, 361, 1205, 363320, 0 UMA Hash: 256, 0, 4, 11, 7, 0 16 Bucket: 152, 0, 45, 30, 72, 0 32 Bucket: 280, 0, 25, 45, 69, 0 64 Bucket: 536, 0, 17, 25, 55, 53 128 Bucket: 1048, 0, 287, 88, 1200, 95423 VM OBJECT: 224, 0, 5536, 23228, 7675004, 0 MAP: 352, 0, 7, 15, 7, 0 KMAP ENTRY: 112, 90222, 283, 1037, 1207524, 0 MAP ENTRY: 112, 0, 1396, 419, 72221561, 0 PV ENTRY: 48, 2244600, 17835, 30261, 768591673, 0 DP fakepg: 120, 0, 0, 31, 10, 0 mt_zone: 1024, 0, 170, 6, 170, 0 16: 16, 0, 3578, 2470, 745206870, 0 32: 32, 0, 1273, 343, 1750850, 0 64: 64, 0, 6147, 1693, 487691440, 0 128: 128, 0, 4659, 387, 1464251, 0 256: 256, 0, 596, 2539, 7208469, 0 512: 512, 0, 608, 253, 791295, 0 1024: 1024, 0, 49, 239, 82867, 0 2048: 2048, 0, 27, 295, 115362, 0 4096: 4096, 0, 240, 278, 564659, 0 Files: 120, 0, 544, 324, 263880246, 0 TURNSTILE: 104, 0, 181, 83, 307, 0 PROC: 856, 0, 82, 82, 308409, 0 THREAD: 608, 0, 169, 11, 24468, 0 KSEGRP: 136, 0, 165, 69, 165, 0 UPCALL: 88, 0, 3, 73, 3, 0 SLEEPQUEUE: 64, 0, 181, 99, 307, 0 VMSPACE: 544, 0, 35, 77, 310929, 0 mbuf_packet: 256, 0, 368, 115, 1331807039, 0 mbuf: 256, 0, 2016, 2331, 5433003167, 0 mbuf_cluster: 2048, 32768, 483, 239, 1236143964, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 216, 0, 4, 410, 48175991, 0 ata_request: 336, 0, 0, 22, 24, 0 ata_composite: 376, 0, 0, 0, 0, 0 VNODE: 496, 0, 28250, 21270, 911708, 0 VNODEPOLL: 152, 0, 0, 0, 0, 0 S VFS Cache: 104, 0, 29153, 9979, 1387950, 0 L VFS Cache: 327, 0, 258, 282, 9423, 0 NAMEI: 1024, 0, 0, 260, 286369405, 0 NFSMOUNT: 584, 0, 1, 6, 1, 0 NFSNODE: 664, 0, 1, 5, 126, 0 DIRHASH: 1024, 0, 278, 122, 1954, 0 PIPE: 768, 0, 35, 335, 253930, 0 KNOTE: 120, 0, 354, 235, 689363256, 0 socket: 616, 49152, 504, 264, 1311349, 0 ipq: 56, 1071, 0, 0, 135, 0 udpcb: 304, 49152, 6, 42, 185368, 0 inpcb: 304, 49152, 384, 192, 903992, 0 tcpcb: 752, 49155, 376, 179, 903992, 0 tcptw: 80, 8235, 8, 487, 211995, 0 syncache: 128, 15370, 0, 145, 890626, 0 hostcache: 136, 15372, 2620, 572, 251887, 0 tcpreass: 40, 2100, 0, 336, 265497, 0 sackhole: 32, 0, 7, 397, 30124600, 0 ripcb: 304, 49152, 0, 36, 64, 0 unpcb: 200, 49153, 40, 340, 221924, 0 rtentry: 264, 0, 6, 36, 26, 0 divcb: 304, 49152, 0, 0, 0, 0 IPFW dynamic rule: 120, 0, 212, 346, 113020, 0 SWAPMETA: 288, 116519, 280, 786, 87016, 0 Mountpoints: 792, 0, 9, 16, 13, 0 FFS inode: 192, 0, 28202, 6158, 911421, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 28202, 5713, 911421, 0 2381/2449/4830 mbufs in use (current/cache/total) 368/354/722/32768 mbuf clusters in use (current/cache/total/max) 368/112 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1331K/1320K/2651K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 8441913 requests for I/O initiated by sendfile 6263 calls to protocol drain routines interrupt total rate irq6: fdc0 8 0 irq14: ata0 47 0 irq16: uhci0 1464547796 1870 irq18: uhci2 12614009 16 irq23: ehci0 3 0 irq46: amr0 12215890 15 irq64: em0 1463513610 1869 cpu0: timer 1564021008 1997 cpu1: timer 1565552539 1999 Total 6082464910 7768 procs memory page disk faults cpu r b w avm fre flt re pi po fr sr am0 in sy cs us sy id 0 2 0 84568 155760 130 3 3 0 298 193 0 771 3421 2251 0 5 95 0 2 0 84576 155488 18 0 0 0 0 0 9 3167 219 7860 0 2 98 0 2 0 84576 155360 0 0 0 0 0 0 2 3568 155 8485 0 1 99 0 2 0 84576 155296 0 0 0 0 0 0 1 2298 110 6218 0 0 100 0 2 0 84576 155232 0 0 0 0 0 0 1 1288 110 4568 0 0 100 0 2 0 84580 154792 1 0 0 0 0 0 10 1459 896 4830 0 1 99 0 2 0 84580 154664 0 0 0 0 0 0 2 2718 128 6911 0 1 99 0 2 0 84580 154376 0 0 0 0 4 0 8 1436 200 4834 0 0 100 0 2 0 84580 154312 0 0 0 0 0 0 1 1500 110 4938 0 0 100 --------------040405020607060803010501 Content-Type: text/plain; name="vmstat-good.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vmstat-good.txt" ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 240, 0, 71, 4, 71, 0 UMA Zones: 376, 0, 71, 9, 71, 0 UMA Slabs: 128, 0, 1003, 70, 237825, 0 UMA RCntSlabs: 128, 0, 502, 2108, 357019, 0 UMA Hash: 256, 0, 4, 11, 7, 0 16 Bucket: 152, 0, 45, 30, 72, 0 32 Bucket: 280, 0, 25, 45, 69, 0 64 Bucket: 536, 0, 17, 25, 55, 53 128 Bucket: 1048, 0, 304, 80, 1200, 95423 VM OBJECT: 224, 0, 4475, 24289, 7583940, 0 MAP: 352, 0, 7, 15, 7, 0 KMAP ENTRY: 112, 90222, 327, 993, 1178293, 0 MAP ENTRY: 112, 0, 1396, 683, 71990087, 0 PV ENTRY: 48, 2244600, 16841, 31255, 764952854, 0 DP fakepg: 120, 0, 0, 31, 10, 0 mt_zone: 1024, 0, 170, 6, 170, 0 16: 16, 0, 3147, 1725, 721283017, 0 32: 32, 0, 1273, 343, 1378831, 0 64: 64, 0, 6161, 1567, 487602322, 0 128: 128, 0, 4658, 388, 1442320, 0 256: 256, 0, 609, 1836, 7119682, 0 512: 512, 0, 608, 253, 781061, 0 1024: 1024, 0, 49, 239, 81907, 0 2048: 2048, 0, 29, 249, 114521, 0 4096: 4096, 0, 239, 294, 558310, 0 Files: 120, 0, 274, 408, 250373577, 0 TURNSTILE: 104, 0, 181, 83, 307, 0 PROC: 856, 0, 82, 82, 304241, 0 THREAD: 608, 0, 169, 11, 24468, 0 KSEGRP: 136, 0, 165, 69, 165, 0 UPCALL: 88, 0, 3, 73, 3, 0 SLEEPQUEUE: 64, 0, 181, 99, 307, 0 VMSPACE: 544, 0, 35, 77, 306723, 0 mbuf_packet: 256, 0, 619, 207, 1297797817, 0 mbuf: 256, 0, 1584, 1190, 5274774672, 0 mbuf_cluster: 2048, 32768, 826, 178, 1203897447, 0 mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0, 0 mbuf_jumbo_9k: 9216, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 0, 0, 0, 0, 0 ACL UMA zone: 388, 0, 0, 0, 0, 0 g_bio: 216, 0, 0, 270, 47261412, 0 ata_request: 336, 0, 0, 22, 24, 0 ata_composite: 376, 0, 0, 0, 0, 0 VNODE: 496, 0, 26256, 23264, 899305, 0 VNODEPOLL: 152, 0, 0, 0, 0, 0 S VFS Cache: 104, 0, 27155, 11977, 1367768, 0 L VFS Cache: 327, 0, 227, 313, 9350, 0 NAMEI: 1024, 0, 0, 260, 272236181, 0 NFSMOUNT: 584, 0, 1, 6, 1, 0 NFSNODE: 664, 0, 1, 5, 126, 0 DIRHASH: 1024, 0, 278, 122, 1938, 0 PIPE: 768, 0, 35, 335, 250212, 0 KNOTE: 120, 0, 93, 372, 666594974, 0 socket: 616, 49152, 212, 466, 1282315, 0 ipq: 56, 1071, 0, 0, 135, 0 udpcb: 304, 49152, 6, 42, 183757, 0 inpcb: 304, 49152, 225, 351, 877750, 0 tcpcb: 752, 49155, 166, 389, 877750, 0 tcptw: 80, 8235, 59, 436, 204414, 0 syncache: 128, 15370, 0, 145, 864539, 0 hostcache: 136, 15372, 2200, 348, 244865, 0 tcpreass: 40, 2100, 0, 336, 252564, 0 sackhole: 32, 0, 20, 384, 29347536, 0 ripcb: 304, 49152, 0, 36, 64, 0 unpcb: 200, 49153, 40, 340, 220743, 0 rtentry: 264, 0, 6, 36, 26, 0 divcb: 304, 49152, 0, 0, 0, 0 IPFW dynamic rule: 120, 0, 215, 219, 112109, 0 SWAPMETA: 288, 116519, 280, 786, 85846, 0 Mountpoints: 792, 0, 9, 16, 13, 0 FFS inode: 192, 0, 26208, 8152, 899018, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 26208, 7722, 899018, 0 2195/1405/3600 mbufs in use (current/cache/total) 619/385/1004/32768 mbuf clusters in use (current/cache/total/max) 619/127 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1786K/1121K/2908K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 8233198 requests for I/O initiated by sendfile 6116 calls to protocol drain routines interrupt total rate irq6: fdc0 8 0 irq14: ata0 47 0 irq16: uhci0 1428187319 1851 irq18: uhci2 12374352 16 irq23: ehci0 3 0 irq46: amr0 11983237 15 irq64: em0 1427141755 1850 cpu0: timer 1540896452 1997 cpu1: timer 1542377798 1999 Total 5962960971 7730 procs memory page disk faults cpu r b w avm fre flt re pi po fr sr am0 in sy cs us sy id 0 1 0 80564 117716 131 3 3 0 298 191 0 628 3371 1994 0 5 95 0 1 0 80568 117640 5 0 0 0 2 0 10 10265 7724 20485 1 6 93 0 1 0 80568 117400 0 0 0 0 0 0 3 10589 8031 21145 0 8 92 0 1 0 80568 116888 0 0 0 0 0 0 8 11745 8362 22538 0 12 88 0 1 0 80568 116312 0 0 0 0 0 0 9 12191 10091 23571 1 11 88 0 1 0 80568 116184 0 0 0 0 0 0 2 13182 10350 25259 1 12 87 0 2 0 80568 115928 0 0 0 0 0 0 3 12896 8176 24112 0 10 90 0 1 0 80568 115736 0 0 0 0 33 0 4 9527 5090 18717 0 9 91 0 1 0 80568 115608 0 0 0 0 32 0 2 13953 11915 26066 0 11 89 --------------040405020607060803010501-- From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 17 22:04:27 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB9F916A420; Wed, 17 Oct 2007 22:04:27 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.freebsd.org (Postfix) with ESMTP id 106BF13C47E; Wed, 17 Oct 2007 22:04:26 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.4/8.13.4) with ESMTP id l9HM4NbF035182; Thu, 18 Oct 2007 02:04:23 +0400 (MSD) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.4/8.13.4/Submit) id l9HM4MWd035173; Thu, 18 Oct 2007 02:04:22 +0400 (MSD) (envelope-from yar) Date: Thu, 18 Oct 2007 02:04:21 +0400 From: Yar Tikhiy To: obrien@freebsd.org, freebsd-hackers@freebsd.org Message-ID: <20071017220421.GD25575@comp.chem.msu.su> References: <20070901073440.GL85633@comp.chem.msu.su> <46DAFE5C.6070806@freebsd.org> <20070903120353.GH30502@comp.chem.msu.su> <200709261028.43378.jhb@freebsd.org> <20071004022344.GA60878@dragon.NUXI.org> <20071013060138.GA14388@comp.chem.msu.su> <20071015173826.GA88628@dragon.NUXI.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20071015173826.GA88628@dragon.NUXI.org> User-Agent: Mutt/1.5.9i Cc: Subject: Re: Useful tools missing from /rescue X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 22:04:27 -0000 On Mon, Oct 15, 2007 at 10:38:26AM -0700, David O'Brien wrote: > On Sat, Oct 13, 2007 at 10:01:39AM +0400, Yar Tikhiy wrote: > > On Wed, Oct 03, 2007 at 07:23:44PM -0700, David O'Brien wrote: > > > I also don't see the need for pgrep - I think needing that says your > > > system is running multiuser pretty well. > > > > First of all, I'd like to point out that /rescue doesn't need to > > be as minimal as /stand used to. Now, /rescue is a compact yet > > versatile set of essential tools that can help in any difficult > > situation when /*bin:/usr/*bin are unusable for some reason, not > > only in restoring a broken system while in single-user mode. A > .. > > As for pgrep+pkill, it can come handy if one has screwed up his > > live system and wants to recover it without dropping the system to > > single-user. > > But if we take this just a little bit farther then why don't we go back > to a static /[s]bin except for the few things one might need LDAP, etc.. > for? That is, what's the purpose in continuing to duplicate /[s]bin > into /rescue? /rescue should be just enough to reasonably get a system > who's shared libs are messed up working again. Note that /rescue includes the most essential tools from /usr/[s]bin, too. Irrespective of its initial purpose, I regard /rescue as an emergency toolset left aside. In particular, it's good to know it's there when you experiment with a live remote system. > > A valid objection to this point is that pgrep's job > > can be done with a combination of ps(1) and sed(1), so it's just a > > matter of convenience. > > I guess I'm still having trouble understanding why one would need 'ps' > to fix a shared libs issue. Now is a reason to keep adding stuff to IMHO it isn't only shared libs issues that /rescue can help with. > /rescue. Also why one would be running 'ps -aux', which is the only way > I can think of to get more than one screen of output if a system is in > trouble. Imagine that you've rm'ed /usr by accident in a remote shell session. With enough tools in /rescue (which doesn't take lots of tools,) you can stop sensitive daemons, find the backup, restore from it, and get a functional system again without a reboot. No doubt, some tools just make the task easier by providing typical command-line idioms. I don't mean I'm so reckless that I need to restore my /usr often, but the 3-4 megabytes occupied by /rescue are a terribly low price today for being able to shoot different parts of one's foot without necessarily hitting the bone. > > The price for it in terms of disk space is next to nothing, and there > > are quite useless space hogs in /rescue already (see below on > > /rescue/vi.) > > Considering how few people are skilled in ed(1) these days, we have > little choice but include vi. Of course, there should be /rescue/vi, and I have an idea on how to remove its dependence on /usr in a more or less elegant way. I mentioned the not-so-functional /rescue/vi here just to show that we can tolerate certain space waste in /rescue. > > I won't speak for everyone, but I really like to use fancy shell > > commands, particularly during hard times: loops, pipelines, etc. > > So I don't have to enter many commands for a single task or browse > > I guess I'm not creative enough in the ways I've screwed up my systems > and needed tools from /rescue. 8-) Just try to installworld FreeBSD/amd64 over a running FreeBSD/i386. ;-) > > > I don't see the purpose of chown - if you have to fall back to /rescue > > > you're user 'root' - and you're trying to fix enough so you can use > > > standard /*lib & /*bin > .. > > Having /rescue/chown is just a matter of completeness of the ch* > > subset of /rescue tools because chown's job can't be done by any > > other stock tools. If /rescue is complete enough, one can find > > more applications for it. E.g., the loader, a kernel, and /rescue > > /rescue wasn't intended to be well orthogonal. /rescue was part of he > corner stone of the deal to switch to shared /[s]bin. But it doesn't confine us to the corner forever. Having an emergency toolset independent of the rest of the system is good in any case. I bet people will experiment and have fun with their systems more eagerly if they know they still can recover quickly with ready tools in case of a serious error. -- Yar From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 02:31:16 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4DCBE16A419 for ; Thu, 18 Oct 2007 02:31:16 +0000 (UTC) (envelope-from brunotm@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.188]) by mx1.freebsd.org (Postfix) with ESMTP id 23A5C13C448 for ; Thu, 18 Oct 2007 02:31:16 +0000 (UTC) (envelope-from brunotm@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so18586rvb for ; Wed, 17 Oct 2007 19:31:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=IhL6eh0BNfqVE7cQmTO8E9kAiMVr3MacEBeoYV9Tf0o=; b=rKmPKpnNhJPJRBJsRdjXonro3LUdM68wPCsfT2OBQK4+Rr9fbU8AndeK6rvZ7dePuz28gSKzuMlnveONtDkbiT8o+62RmOd530MOtDcEzPKpP9qnYyveHxnYSdyrH5ProgHFcYXnaS8dz9TZ9De/iePR700u5NIKfBMQrhXj3G0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=RVZl0Jv57Q18QxZnC4uabPLXM5rrD8ZmgHTB5qlnK+nk/RGLXxynBLX8eNqXRHgWd/TcRh0hZGTb15SD8EElxT8ToZbhqIuk1Omg6q8Ho89gId6VsXO25OPJ6+YSuUnD7+AJyDSeFkBrI+LoCbKo3efZvBw+BobcPrvRxxHGuJ0= Received: by 10.114.197.1 with SMTP id u1mr2648waf.1192672940033; Wed, 17 Oct 2007 19:02:20 -0700 (PDT) Received: by 10.114.56.5 with HTTP; Wed, 17 Oct 2007 19:02:19 -0700 (PDT) Message-ID: <787586d00710171902o59c82171y1e972ff51c2174b6@mail.gmail.com> Date: Thu, 18 Oct 2007 00:02:19 -0200 From: "Bruno Tavares" To: "Ulf Lilleengen" , freebsd-hackers@freebsd.org, s223560@studenti.ing.unipi.it, luigi@freebsd.org In-Reply-To: <20071017143954.GU99087@gandalf.sssup.it> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> <20071017110715.GA25075@stud.ntnu.no> <20071017121907.GL99087@gandalf.sssup.it> <20071017130934.GA26180@stud.ntnu.no> <20071017143954.GU99087@gandalf.sssup.it> Cc: Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 02:31:16 -0000 On 10/17/07, Fabio Checconi wrote: > > From: Ulf Lilleengen > > Date: Wed, Oct 17, 2007 03:09:35PM +0200 > > > > On ons, okt 17, 2007 at 02:19:07 +0200, Fabio Checconi wrote: > > > Maybe I've not enough experience/knowledge of the driver subsystem, > [...] > > If you look at it, Hybrid is just a generalization of the existing > > bioq_* API already defined. And this API is used by GEOM classes _before_ > > device drivers get the requests AFAIK. > > > > I looked at the Hybrid code, but I don't think that the bioq_* > family of calls can be the right place to start, for the problems > experienced during the Hybrid development with locking/anticipation > and because you can have the same request passing through multiple > bioqs during its path to the device (e.g., two stacked geoms using > two different bioqs and then a device driver using bioq_* to organize > its queue, or geoms using more than one bioq, like raid3; I think > the complexity can become unmanageable.) One could even think to > configure each single bioq in the system, but things can get very > complex in this way. > > > > For a simple example on a driver, the md-driver might be a good place to > > look. Note that I have little experience and knowledge of the driver > > subsystem myself. > > > > I'll take a look, thanks. > > > > Also note (from the Hybrid page): > > * we could not provide support for non work-conserving schedulers, due to a > [...] > > > > This certainly argues for having this in the GEOM layer, but perhaps it's > > possible to change the assumtions done in some drivers? The locking issue > > should perhaps be better planned though, and an audit of the driver disksort > > code is necessary. > > > > I need some more time to think about that :) > > > > Also: > > * as said, the ATA driver in 6.x/7.x moves the disksort one layer below the > > one we are working at, so this particular work won't help on ATA-based 6.x > > machines. > > We should figure out how to address this, because the work done at that > > layer is mostly a replica of the bioq_*() API. > > > > So, I see this can get a bit messy thinking of that the ATA drivers does > > disksorts on its own, but perhaps it would be possible to fix this by letting > > changing the general ATA driver to use it's own pluggable scheduler. > > > > Anyway, I shouldn't demand that you do this, especially since I don't have > > any code or anything to show to, and because you decide what you want to do. > > I still cannot say if a GEOM scheduler is better than a scheduler > put at a lower level, or if the bioq_* interface is better than any > other alternative, so your suggestions are welcome. Moreover I'd > really like to discuss/work together, or at least do things with > some agreement on them. If I'll have the time to experiment with > more than one solution I'll be happy to do that. > > > > However, I'd hate to see the Hybrid effort go to waste :) I was hoping some > > of the authors of the project would reply with their thoughts, so I CC'ed > > them. > > Well, the work done on Hybrid had also interesting aspects from > the algorithm side... but that's another story... > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- This .signature sanitized for your protection From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 12:01:45 2007 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4CF3816A418 for ; Thu, 18 Oct 2007 12:01:45 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from cs1.cs.huji.ac.il (cs1.cs.huji.ac.il [132.65.16.10]) by mx1.freebsd.org (Postfix) with ESMTP id E8E2313C50E for ; Thu, 18 Oct 2007 12:01:44 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from pampa.cs.huji.ac.il ([132.65.80.32]) by cs1.cs.huji.ac.il with esmtp id 1IiTeh-0007tt-G6; Thu, 18 Oct 2007 13:35:11 +0200 X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.2 To: hackers@FreeBSD.org In-reply-to: References: <20070904233246.GA2409@epia-2.farid-hajji.net> <043a01c7f202$a7ad0920$f7071b60$@co.uk> <046801c7f229$a4534510$ecf9cf30$@co.uk> Comments: In-reply-to Danny Braniss message dated "Mon, 10 Sep 2007 11:14:45 +0300." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 18 Oct 2007 13:35:11 +0200 From: Danny Braniss Message-ID: Cc: freebsd-current@FreeBSD.org Subject: Re: dump problems X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 12:01:45 -0000 > > > > > > > > the only indication I can see, is that one of the dump procs. is > > > > waiting on > > > > sbwait, and probably it's some deadlock, which is similar to what I > > > > keep > > > > seeing here, i'll try now with SCHED_ULE to see if it make a > > > > difference. > > > > > > > > > I'm running SCHED_ULE on these already, if your not I guess it's not the > > > scheduler? > > > > > I can get it to 'work' by fiddling with the b flag (blocksize), which still > > points to some timming/deadlock problem. > > ie: > > dump 0abf 64 /some/backup/file /file/to/backup > > now works, but > > dump 0abf 64 - | restore rbf 64 - > > hangs as before. (i don't think the b 64 in restore is needed). > > ok, it is time to look at the sources, this program has been around since the > beginin of time, > or at least since Unix V6 :-), and it has been hacked ever since. but now that > most of you never heard of 9track tapes, etc, I was wondering if there is a > point in hacking at > it again. > pros: dump/restore has never failed me till now. > cons: there are other programs tar/cpio/gtar/etc, but they each have their > nits. > so here are some questions: > - is the readers/writer split realy needed now? my guess it was put in in the > old > days to get tapes streaming - which is btw, what's not working. > - is it/will it be needed for ZFS? [dump is for ufs ...] > > danny Assumption: Since dump was written, much has changed, and in the days of single-core/UP the Caltech hack (see /usr/src/sbin/dump/tape.c) works. On newer multicore hardware it will enter into deadlock and thus hang, specially if 'tape' is a file or pipe. While I'm now putting on the surgical cap to dissect tape.c, any further insight is welcome. cheers, danny From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 16:16:11 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC98416A417 for ; Thu, 18 Oct 2007 16:16:11 +0000 (UTC) (envelope-from dclements@gmail.com) Received: from rn-out-0102.google.com (rn-out-0910.google.com [64.233.170.191]) by mx1.freebsd.org (Postfix) with ESMTP id 3566313C474 for ; Thu, 18 Oct 2007 16:16:10 +0000 (UTC) (envelope-from dclements@gmail.com) Received: by rn-out-0102.google.com with SMTP id s42so109967rnb for ; Thu, 18 Oct 2007 09:16:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=lbeDxgzdLoOa4cTk92ktaO9vLsB0609wga9IT/BGKuA=; b=ElUwFs24RyakHfRZD9Rl3rfctdfBCVEoH1ckIgIARttEaW5jq697omokiadPmgh1jmn1k0fl3SuNJRervQlZ2l4TDXEX0UTbPuRYg+7C0eSmX0lle9u5JNYX970x5dWm6GK2XMtRhL0tH0vgUWrAM3ArSTOM9OLP9TNe6N5Rt04= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=cULW4ADl0E3YZclPfu8GJwM76JlWb+IIiAxNV4ZE/ojkvhms6PW7s3WqInid1UtCQvS+b7+UXbrCACHWqGNqPN1gmUNqh1VzLLTt2wdEXCdskeceRRvIHCkNAwZ/D+ouUd3naK3yyPuzx7yxzYtFgPe0RtbeUCPMQw6F8quC06U= Received: by 10.142.99.21 with SMTP id w21mr254924wfb.1192724169317; Thu, 18 Oct 2007 09:16:09 -0700 (PDT) Received: by 10.142.81.19 with HTTP; Thu, 18 Oct 2007 09:16:09 -0700 (PDT) Message-ID: <54da514e0710180916i2b616977p6cf27740a5d9a997@mail.gmail.com> Date: Thu, 18 Oct 2007 09:16:09 -0700 From: "Doug Clements" To: freebsd-hackers@freebsd.org In-Reply-To: <54da514e0710170931r1b558bedi69e1bba50807f465@mail.gmail.com> MIME-Version: 1.0 References: <54da514e0710151003o684dbc9dle0244b9d1ca0528f@mail.gmail.com> <54da514e0710151017t5e12e404uf1258fceeaa3f149@mail.gmail.com> <4715F1A5.50605@freebsd.org> <54da514e0710170931r1b558bedi69e1bba50807f465@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Interrupt/speed problems with 6.2 NFS server X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 16:16:11 -0000 On 10/17/07, Doug Clements wrote: > > Can you also send the output of ps -auxl? > > > > Also - do you notice this performance drop when running something like > > one of the network performance tools? I'd like to isolate the disk > > activity from the network activity for a clean test.. > > > > I tested this with iperf, and while I did see some nfs performance > degradation, it did not bog the machine or delay the terminal in the same > way. NFS requests were still processed in an acceptable fashion. It was > still responsive to commands and ran processes just fine. I would have to > say it performed as to be expected during iperf tests (on which I got about > 85mbit/sec, which is also to be expected). Interrupts went down to about > 2000/sec on em, but the machine did not hang. > > Something I've noticed is that running 'systat -vm' seems to be part of > the problem. If I run the file copy by itself with rdist, it's fast and runs > ok. If I run it with systat -vm going, this is when the interrupts jump way > up and the machine starts to delay badly. Noticing this, I tried running > 'sysctl -a' during the file copy, thinking there was some problem with > polling the kernel for certain statistics. Sure enough, sysctl -a delays at > 2 spots. Once right after " kern.random.sys.harvest.swi: 0" and once again > after "debug.hashstat.nchash: 131072 61777 6 4713". While it is delayed > here (for a couple seconds for each one) the machine is totally hung. Maybe > this is a statistics polling issue? Maybe the machine is delayed just long > enough in systat -vm to make the nfs clients retry, causing a storm of > interrupts? > > Other systat modes do not seem to cause the same problem (pigs, icmp, > ifstat). > > I do not think the ps or systat output is very accurate, since I can't get > them to run when the machine is hung up. I type in the command, but it does > not run until the machine springs back to life. I'm not sure how this will > affect measurements. > > http://toric.loungenet.org/~doug/sysctl-a > http://toric.loungenet.org/~doug/psauxl > http://toric.loungenet.org/~doug/systat-vm > > My real confusion lies in why there are still em interrupts at all, with > polling on. > > Thanks! > > --Doug > Well, I'm an idiot. I for some reason though SMP was default in GENERIC, and didn't really think to check. This is a quad-core box, so on top of POLLING, I enabled SMP. Now, polling seems to work great (I have no interrupts on the em device), only the 2000 interrupts/sec on each CPU. And my strange delay is gone. "systat -vm" runs great, "sysctl -a" can be run over and over again with no lag and no slowness. I can even run concurrent backups (da1 -> da2, and da2 -> da1). Am I missing something in that SMP is a requirement for POLLING? I've never come across that before. --Doug Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #1: Tue Oct 16 16:45:43 PDT 2007 admin@madonna.linkline.com:/usr/obj/usr/src/sys/MADONNA ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4 Features=0xbfebfbff Features2=0x4e3bd,CX16,,,> AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 2 real memory = 4831838208 (4608 MB) avail memory = 4124962816 (3933 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_throttle0: on cpu0 cpu1: on acpi0 acpi_throttle1: on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 cpu2: on acpi0 acpi_throttle2: on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 cpu3: on acpi0 acpi_throttle3: on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 acpi_button0: on acpi0 acpi_button1: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at device 0.0 on pci5 em0: Ethernet address: 00:15:17:21:bf:30 em1: port 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at device 0.1 on pci5 em1: Ethernet address: 00:15:17:21:bf:31 pcib6: at device 0.3 on pci1 pci6: on pcib6 3ware device driver for 9000 series storage controllers, version: 3.60.02.012 twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6 twa0: [FAST] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports, Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002 pcib7: at device 3.0 on pci0 pci7: on pcib7 pcib8: at device 4.0 on pci0 pci8: on pcib8 pcib9: at device 5.0 on pci0 pci9: on pcib9 pcib10: at device 6.0 on pci0 pci10: on pcib10 pcib11: at device 7.0 on pci0 pci11: on pcib11 pci0: at device 8.0 (no driver attached) pcib12: irq 16 at device 28.0 on pci0 pci12: on pcib12 uhci0: port 0x4080-0x409f irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] usb0: on uhci0 usb0: USB revision 1.0 uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4060-0x407f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] usb1: on uhci1 usb1: USB revision 1.0 uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4040-0x405f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] usb2: on uhci2 usb2: USB revision 1.0 uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4020-0x403f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] usb3: on uhci3 usb3: USB revision 1.0 uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xf8c00400-0xf8c007ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub4: 8 ports with 8 removable, self powered pcib13: at device 30.0 on pci0 pci13: on pcib13 pci13: at device 12.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0 ata2: on atapci1 ata3: on atapci1 pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: CDRW at ata0-master UDMA33 da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C) da1 at twa0 bus 0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) da2 at twa0 bus 0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C) lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic6: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic7: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/da0s1a em1: link state changed to UP From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 20:27:43 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 398E116A469; Thu, 18 Oct 2007 20:27:43 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from mail.kuban.ru (mail.kuban.ru [62.183.66.246]) by mx1.freebsd.org (Postfix) with ESMTP id 96FCE13C4B6; Thu, 18 Oct 2007 20:27:42 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from localhost.my.domain ([85.172.12.98]) by mail.kuban.ru (8.9.1/8.9.1) with ESMTP id l9IJrKx1086473; Thu, 18 Oct 2007 23:53:33 +0400 (MSD) Received: (from bsam@localhost) by localhost.my.domain (8.14.1/8.14.1/Submit) id l9IJuQCr023936; Thu, 18 Oct 2007 23:56:26 +0400 (MSD) (envelope-from bsam@ipt.ru) X-Authentication-Warning: localhost.my.domain: bsam set sender to bsam@ipt.ru using -f References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> <47165A01.1030806@chistydom.ru> From: Boris Samorodov In-Reply-To: <47165A01.1030806@chistydom.ru> (Alexey Popov's message of "Wed\, 17 Oct 2007 22\:52\:49 +0400") User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (berkeley-unix) Date: Thu, 18 Oct 2007 23:56:26 +0400 Message-ID: <07289061@ipt.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Thu, 18 Oct 2007 20:43:32 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 20:27:43 -0000 Hi! Since nobody answered so far, here is my two cents. I'm not an expert here so it's only my imho. On Wed, 17 Oct 2007 22:52:49 +0400 Alexey Popov wrote: > interrupt total rate > irq6: fdc0 8 0 > irq14: ata0 47 0 > irq16: uhci0 1428187319 1851 ^^^^^^^^^^ ^^^^ [1] > irq18: uhci2 12374352 16 > irq23: ehci0 3 0 > irq46: amr0 11983237 15 > irq64: em0 1427141755 1850 ^^^^^^^^^^ ^^^^ [2] > cpu0: timer 1540896452 1997 > cpu1: timer 1542377798 1999 > Total 5962960971 7730 [1] and [2] looks suspicious to me (totals and rate are too close to each other and btw to timers). Let the latter (timers) alone. Do you use any USB device? Can you try to use other network card? That behaviour seems to be an interrupt storm and/or irq collision. WBR -- bsam From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 22:15:45 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE06F16A475 for ; Thu, 18 Oct 2007 22:15:45 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 48C2713C4A8 for ; Thu, 18 Oct 2007 22:15:45 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id l9ILvVaw004151; Thu, 18 Oct 2007 15:57:32 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4717D6BC.5090206@samsco.org> Date: Thu, 18 Oct 2007 15:57:16 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4 MIME-Version: 1.0 To: Boris Samorodov References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> <47165A01.1030806@chistydom.ru> <07289061@ipt.ru> In-Reply-To: <07289061@ipt.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Thu, 18 Oct 2007 15:57:32 -0600 (MDT) X-Spam-Status: No, score=-1.4 required=5.5 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org X-Mailman-Approved-At: Fri, 19 Oct 2007 03:08:53 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 22:15:45 -0000 Boris Samorodov wrote: > Hi! > > Since nobody answered so far, here is my two cents. I'm not an expert > here so it's only my imho. > > On Wed, 17 Oct 2007 22:52:49 +0400 Alexey Popov wrote: > >> interrupt total rate >> irq6: fdc0 8 0 >> irq14: ata0 47 0 >> irq16: uhci0 1428187319 1851 > ^^^^^^^^^^ ^^^^ [1] >> irq18: uhci2 12374352 16 >> irq23: ehci0 3 0 >> irq46: amr0 11983237 15 >> irq64: em0 1427141755 1850 > ^^^^^^^^^^ ^^^^ [2] >> cpu0: timer 1540896452 1997 >> cpu1: timer 1542377798 1999 >> Total 5962960971 7730 > > [1] and [2] looks suspicious to me (totals and rate are too close to > each other and btw to timers). Let the latter (timers) alone. Do you > use any USB device? Can you try to use other network card? That > behaviour seems to be an interrupt storm and/or irq collision. > > It's neither. It's a side effect of a feature that FreeBSD abuses for handling interrupts. Note that amr0 and ehci2 are acting similar. It's mostly harmless, but it does waste CPU cycles. I wouldn't expect this on a recent version of FreeBSD, though, at least not from the e1000 driver. Scott From owner-freebsd-hackers@FreeBSD.ORG Thu Oct 18 23:40:46 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57A3B16A421; Thu, 18 Oct 2007 23:40:46 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from services.ipt.ru (services.ipt.ru [194.62.233.110]) by mx1.freebsd.org (Postfix) with ESMTP id 0848513C4B9; Thu, 18 Oct 2007 23:40:45 +0000 (UTC) (envelope-from bsam@ipt.ru) Received: from [85.172.12.98] (helo=localhost.my.domain) by services.ipt.ru with esmtpa (Exim 4.54 (FreeBSD)) id 1IieJr-0008B5-Nu; Fri, 19 Oct 2007 02:58:23 +0400 To: Scott Long References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> <47165A01.1030806@chistydom.ru> <07289061@ipt.ru> <4717D6BC.5090206@samsco.org> From: Boris Samorodov Date: Fri, 19 Oct 2007 03:00:52 +0400 In-Reply-To: <4717D6BC.5090206@samsco.org> (Scott Long's message of "Thu\, 18 Oct 2007 15\:57\:16 -0600") Message-ID: <32246027@ipt.ru> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailman-Approved-At: Fri, 19 Oct 2007 03:08:53 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2007 23:40:46 -0000 On Thu, 18 Oct 2007 15:57:16 -0600 Scott Long wrote: > Boris Samorodov wrote: > > Since nobody answered so far, here is my two cents. I'm not an expert > > here so it's only my imho. > > > > On Wed, 17 Oct 2007 22:52:49 +0400 Alexey Popov wrote: > > > >> interrupt total rate > >> irq6: fdc0 8 0 > >> irq14: ata0 47 0 > >> irq16: uhci0 1428187319 1851 > > ^^^^^^^^^^ ^^^^ [1] > >> irq18: uhci2 12374352 16 > >> irq23: ehci0 3 0 > >> irq46: amr0 11983237 15 > >> irq64: em0 1427141755 1850 > > ^^^^^^^^^^ ^^^^ [2] > >> cpu0: timer 1540896452 1997 > >> cpu1: timer 1542377798 1999 > >> Total 5962960971 7730 > > > > [1] and [2] looks suspicious to me (totals and rate are too close to > > each other and btw to timers). Let the latter (timers) alone. Do you > > use any USB device? Can you try to use other network card? That > > behaviour seems to be an interrupt storm and/or irq collision. > It's neither. It's a side effect of a feature that FreeBSD abuses for > handling interrupts. Note that amr0 and ehci2 are acting similar. It's > mostly harmless, but it does waste CPU cycles. I wouldn't expect this > on a recent version of FreeBSD, though, at least not from the e1000 > driver. I see. Sorry for the noise. So, as I can understand _that_ can't be the problem (as at subj) the OP is seeing? WBR -- bsam From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 04:56:56 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B944716A41B; Fri, 19 Oct 2007 04:56:56 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id 821C413C459; Fri, 19 Oct 2007 04:56:56 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id 7CF351CC41; Fri, 19 Oct 2007 06:56:54 +0200 (CEST) Date: Fri, 19 Oct 2007 06:56:54 +0200 From: Ed Schouten To: John Baldwin Message-ID: <20071019045654.GF5411@hoeg.nl> References: <20071016094118.GE5411@hoeg.nl> <200710170916.18788.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6Fzmsq5qk3/aYyX1" Content-Disposition: inline In-Reply-To: <200710170916.18788.jhb@freebsd.org> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: FreeBSD Hackers Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 04:56:56 -0000 --6Fzmsq5qk3/aYyX1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * John Baldwin wrote: > The best option right now is to read the code. There are some comments in > both the headers and implementation. Would it be useful to write manpages for these interfaces, or do we assume that only godlike people can use them anyway? I am willing to write manpages for them. Yours, --=20 Ed Schouten WWW: http://g-rave.nl/ --6Fzmsq5qk3/aYyX1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD4DBQFHGDkW52SDGA2eCwURAg1oAJ9sw0L+SUOGnoyGcNr1eR4MROD1YACYsuNd gzPIoApLv0SZqRjuty8NzA== =53Mj -----END PGP SIGNATURE----- --6Fzmsq5qk3/aYyX1-- From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 06:26:30 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 223DE16A417; Fri, 19 Oct 2007 06:26:30 +0000 (UTC) (envelope-from lol@chistydom.ru) Received: from comtv.ru (comtv.ru [217.10.32.17]) by mx1.freebsd.org (Postfix) with ESMTP id 70E8713C455; Fri, 19 Oct 2007 06:26:28 +0000 (UTC) (envelope-from lol@chistydom.ru) X-UCL: actv Received: from yoda.org.ru ([83.167.98.162] verified) by comtv.ru (CommuniGate Pro SMTP 4.1.8) with ESMTP id 7103082; Fri, 19 Oct 2007 10:26:27 +0400 Received: from [80.68.244.40] (adm40.relax.ru [80.68.244.40]) (Authenticated sender: llp@soekris.ru) by yoda.org.ru (Postfix) with ESMTP id 0CE1128BDA; Fri, 19 Oct 2007 10:26:32 +0400 (MSD) Message-ID: <47184DD0.6050704@chistydom.ru> Date: Fri, 19 Oct 2007 10:25:20 +0400 From: Alexey Popov User-Agent: Thunderbird 2.0.0.6 (X11/20070924) MIME-Version: 1.0 To: Scott Long References: <47137D36.1020305@chistydom.ru> <47140906.2020107@FreeBSD.org> <47146FB4.6040306@chistydom.ru> <47147E49.9020301@FreeBSD.org> <47149E6E.9000500@chistydom.ru> <4715035D.2090802@FreeBSD.org> <4715C297.1020905@chistydom.ru> <4715C5D7.7060806@FreeBSD.org> <47165A01.1030806@chistydom.ru> <07289061@ipt.ru> <4717D6BC.5090206@samsco.org> In-Reply-To: <4717D6BC.5090206@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 19 Oct 2007 11:32:03 +0000 Cc: Boris Samorodov , freebsd-hackers@freebsd.org, freebsd-stable@freebsd.org Subject: Re: amrd disk performance drop after running under high load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 06:26:30 -0000 Hi Scott Long wrote: >>> interrupt total rate >>> irq6: fdc0 8 0 >>> irq14: ata0 47 0 >>> irq16: uhci0 1428187319 1851 >> ^^^^^^^^^^ ^^^^ [1] >>> irq18: uhci2 12374352 16 >>> irq23: ehci0 3 0 >>> irq46: amr0 11983237 15 >>> irq64: em0 1427141755 1850 >> ^^^^^^^^^^ ^^^^ [2] >>> cpu0: timer 1540896452 1997 >>> cpu1: timer 1542377798 1999 >>> Total 5962960971 7730 >> >> [1] and [2] looks suspicious to me (totals and rate are too close to >> each other and btw to timers). Let the latter (timers) alone. Do you >> use any USB device? Can you try to use other network card? That >> behaviour seems to be an interrupt storm and/or irq collision. > > It's neither. It's a side effect of a feature that FreeBSD abuses for > handling interrupts. Note that amr0 and ehci2 are acting similar. It's > mostly harmless, but it does waste CPU cycles. I wouldn't expect this > on a recent version of FreeBSD, though, at least not from the e1000 > driver. I have this effect on many servers and I believe it is harmless. At once I was trying to reduce CPU usage on the very loaded server and removed USB from kernel. This effect disappeared, but there was no significant difference in CPU usage. I disagree about your words about recent version. I have this effect on many servers with latest FreeBSD-6-stable and em. Actually I have more servers with this effect than without it. With best regards, Alexey Popov From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 13:51:05 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBB7416A421 for ; Fri, 19 Oct 2007 13:51:05 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id CE76113C49D for ; Fri, 19 Oct 2007 13:51:05 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from zion.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by elvis.mu.org (Postfix) with ESMTP id B0E541A4D8E; Fri, 19 Oct 2007 06:51:04 -0700 (PDT) From: John Baldwin To: Ed Schouten Date: Fri, 19 Oct 2007 08:42:33 -0400 User-Agent: KMail/1.9.7 References: <20071016094118.GE5411@hoeg.nl> <200710170916.18788.jhb@freebsd.org> <20071019045654.GF5411@hoeg.nl> In-Reply-To: <20071019045654.GF5411@hoeg.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200710190842.34286.jhb@freebsd.org> Cc: FreeBSD Hackers Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 13:51:05 -0000 On Friday 19 October 2007 12:56:54 am Ed Schouten wrote: > * John Baldwin wrote: > > The best option right now is to read the code. There are some comments in > > both the headers and implementation. > > Would it be useful to write manpages for these interfaces, or do we > assume that only godlike people can use them anyway? I am willing to > write manpages for them. They already exist, but they really are only used to implement higher-level primitives like locks and condition variables. The rest of the kernel should use the higher-level primitives anyway. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 15:10:06 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C98A16A49A for ; Fri, 19 Oct 2007 15:10:06 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from magellan.palisadesys.com (magellan.palisadesys.com [192.188.162.211]) by mx1.freebsd.org (Postfix) with ESMTP id D0C8313C45D for ; Fri, 19 Oct 2007 15:10:05 +0000 (UTC) (envelope-from ghelmer@palisadesys.com) Received: from [172.16.2.242] (cetus.palisadesys.com [192.188.162.7]) (authenticated bits=0) by magellan.palisadesys.com (8.14.1/8.14.1) with ESMTP id l9JEoGaQ016345 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 19 Oct 2007 09:50:16 -0500 (CDT) (envelope-from ghelmer@palisadesys.com) Message-ID: <4718C43B.2040504@palisadesys.com> Date: Fri, 19 Oct 2007 09:50:35 -0500 From: Guy Helmer User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-3.0 (magellan.palisadesys.com [192.188.162.5]); Fri, 19 Oct 2007 09:50:16 -0500 (CDT) X-Palisade-MailScanner-Information: Please contact the ISP for more information X-Palisade-MailScanner: Found to be clean X-Palisade-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-3.799, required 6, ALL_TRUSTED -1.80, BAYES_00 -2.60, J_CHICKENPOX_53 0.60) X-Palisade-MailScanner-From: ghelmer@palisadesys.com Subject: Netgraph ng_bridge status message missing last three values? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 15:10:06 -0000 I'm diagnosing a system that uses the Netgraph ng_bridge node to bridge two Ethernet segments and the system seems to stop passing traffic at randomly-triggered points (somewhat annoying for a system half a world away behind a very restrictive firewall :-) I've installed a cron job to dump the Netgraph bridge status messages to a log file every few minutes. However, I've found that the ng_bridge status message looks like this: > ngctl msg bnet0: getstats 0 STATS=Rec'd response "getstats" (4) from "[10]:": Args: { recvOctets=474229151 recvPackets=3322699 recvMulticast=128640 recvBroadcast=105813 recvUnknown=2372 xmitOctets=516954216 xmitPackets=2968420 xmitMulticasts=8 xmitBroadcasts=339 } but to aid diagnosis I would really like to see several values that, according to ng_bridge.h, are in the stats structure, including recvRunts, recvInvalid, loopDrops, loopDetects, and memoryFailures. It looks like all the code is there to report the values of these variables, but I can't seem to find anything that is dropping these variables from ngctl's output. Any help is greatly appreciated, Guy -- Guy Helmer, Ph.D. Chief System Architect Palisade Systems, Inc. From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 17:02:25 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDDA316A496 for ; Fri, 19 Oct 2007 17:02:25 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outD.internet-mail-service.net (outD.internet-mail-service.net [216.240.47.227]) by mx1.freebsd.org (Postfix) with ESMTP id B1E3B13C459 for ; Fri, 19 Oct 2007 17:02:23 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 19 Oct 2007 10:02:06 -0700 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 63E96126757; Fri, 19 Oct 2007 10:02:06 -0700 (PDT) Message-ID: <4718E324.9000209@elischer.org> Date: Fri, 19 Oct 2007 10:02:28 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Ed Schouten References: <20071016094118.GE5411@hoeg.nl> <200710170916.18788.jhb@freebsd.org> <20071019045654.GF5411@hoeg.nl> In-Reply-To: <20071019045654.GF5411@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 17:02:25 -0000 Ed Schouten wrote: > * John Baldwin wrote: >> The best option right now is to read the code. There are some comments in >> both the headers and implementation. > > Would it be useful to write manpages for these interfaces, or do we > assume that only godlike people can use them anyway? I am willing to > write manpages for them. > > Yours, From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 17:07:03 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BA0816A418 for ; Fri, 19 Oct 2007 17:07:03 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outB.internet-mail-service.net (outB.internet-mail-service.net [216.240.47.225]) by mx1.freebsd.org (Postfix) with ESMTP id 6D60713C478 for ; Fri, 19 Oct 2007 17:06:59 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 19 Oct 2007 10:06:56 -0700 X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e X-Client-Authorized: MaGic Cook1e Received: from julian-mac.elischer.org (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id 832DD126757; Fri, 19 Oct 2007 10:06:55 -0700 (PDT) Message-ID: <4718E43F.1020303@elischer.org> Date: Fri, 19 Oct 2007 10:07:11 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Ed Schouten References: <20071016094118.GE5411@hoeg.nl> <200710170916.18788.jhb@freebsd.org> <20071019045654.GF5411@hoeg.nl> In-Reply-To: <20071019045654.GF5411@hoeg.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 17:07:03 -0000 Ed Schouten wrote: > * John Baldwin wrote: >> The best option right now is to read the code. There are some comments in >> both the headers and implementation. > > Would it be useful to write manpages for these interfaces, or do we > assume that only godlike people can use them anyway? I am willing to > write manpages for them. > > Yours, man pages are ALWAYS a good idea. (well, NEARLY always.) it may be that this information may go well into an exisiting man page, or at least a description. I believe that to some extent they are supposed to be an internal interface so maybe a description as part of the utilities that use them may be in order.. also think about whether a paragraph might go into locking(9). I'm still trying to nag john to run an editing pass over locking(9) From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 18:09:07 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D196116A417; Fri, 19 Oct 2007 18:09:07 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail6.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 5F4E813C467; Fri, 19 Oct 2007 18:09:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8p) with ESMTP id 215186663-1834499 for multiple; Fri, 19 Oct 2007 14:11:20 -0400 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.13.8/8.13.8) with ESMTP id l9JI8pIb032709; Fri, 19 Oct 2007 14:09:00 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: "Attilio Rao" Date: Fri, 19 Oct 2007 14:05:26 -0400 User-Agent: KMail/1.9.6 References: <20071016094118.GE5411@hoeg.nl> <200710190842.34286.jhb@freebsd.org> <3bbf2fe10710191008r6dc1939em85f8574e107af48b@mail.gmail.com> In-Reply-To: <3bbf2fe10710191008r6dc1939em85f8574e107af48b@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200710191405.26488.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Fri, 19 Oct 2007 14:09:01 -0400 (EDT) X-Virus-Scanned: ClamAV 0.88.3/4545/Wed Oct 17 17:05:57 2007 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: FreeBSD Hackers , Ed Schouten Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 18:09:07 -0000 On Friday 19 October 2007 01:08:27 pm Attilio Rao wrote: > 2007/10/19, John Baldwin : > > On Friday 19 October 2007 12:56:54 am Ed Schouten wrote: > > > * John Baldwin wrote: > > > > The best option right now is to read the code. There are some comments in > > > > both the headers and implementation. > > > > > > Would it be useful to write manpages for these interfaces, or do we > > > assume that only godlike people can use them anyway? I am willing to > > > write manpages for them. > > > > They already exist, but they really are only used to implement higher-level > > primitives like locks and condition variables. The rest of the kernel should > > use the higher-level primitives anyway. > > Well, really turnstiles don't have manpages, but this is still OK as > they are only used in mutex while the "real" sleeping primitive should > be identified by sleepqueues. Ah, there is a sleepqueue(9), but not a turnstile(9). -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 18:21:55 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E22D16A420 for ; Fri, 19 Oct 2007 18:21:55 +0000 (UTC) (envelope-from BearPerson@gmx.net) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.freebsd.org (Postfix) with SMTP id C47A013C4C1 for ; Fri, 19 Oct 2007 18:21:52 +0000 (UTC) (envelope-from BearPerson@gmx.net) Received: (qmail invoked by alias); 19 Oct 2007 18:21:50 -0000 Received: from port-83-236-56-222.dynamic.qsc.de (EHLO gmx.net) [83.236.56.222] by mail.gmx.net (mp010) with SMTP; 19 Oct 2007 20:21:50 +0200 X-Authenticated: #20254835 X-Provags-ID: V01U2FsdGVkX1/npUx3NW/nPcxPTGI/POWeCz027UDPWbPrxt/cFg RJryDif/ceajvr Date: Fri, 19 Oct 2007 20:21:46 +0200 From: Karsten Behrmann To: freebsd-hackers@freebsd.org Message-ID: <20071019202146.7a2cc376@Karsten.Behrmanns.Kasten> In-Reply-To: <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> References: <20071011022001.GC13480@gandalf.sssup.it> <20071016161037.5ab1b74f@39-25.mops.rwth-aachen.de> X-Mailer: Claws Mail 2.9.2 (GTK+ 2.8.18; i686-pc-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_u4LaQi=.EbdLlgeI3g._33m"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-Y-GMX-Trusted: 0 Subject: Re: Pluggable Disk Scheduler Project X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 18:21:55 -0000 --Sig_u4LaQi=.EbdLlgeI3g._33m Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable > I wouldn't focus on just anticipation, but also other types of > schedulers (I/O scheduling influenced by nice value?) One note on this subject: it will be nice for things like bg-fsck, but it also brings up a point that has been overlooked so far: priority propagation. As far as I am aware, there's plenty of locks that a process can hold while waiting for I/O to complete (directories, snapshots, probably some others I missed) Now, normally when a high-priority thread needs a lock owned by another process, it'll bump up that thread's priority and yield, hopefully freeing the lock up quickly. This is obviously not quite so easy for IO. I haven't quite understood the code involved yet, as far as I can tell turnstiles panic when the lock-owning thread is sleeping. What we'll probably need to do is make priority propagation wake up a waiting-for-io process, which then needs to dig up its IO request (which may be anywhere in geom, but potentially held back by the scheduler) and make sure it's [re]queued with higher priority. If we don't do this, we'll get funny effects with a bg-fsck blocking some high-priority process indefinitely because it happens to be waiting for IO and holding the snapshot lock, on an IO-busy system. If we do this, we may get into significant fun with cutting into geom to allow requeuing, or waste some cpu with polling from the queuing geom. This point may not be immediately obvious to people coming from the IO/filesystem field, but it is something we should keep in mind. So Far, Karsten Behrmann --Sig_u4LaQi=.EbdLlgeI3g._33m Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFHGPW9AksKLoO3vywRAh8RAJ9r8zME85G1X7MyJGmL6SxQmPaHMQCdEs8T 6Mpn37GtxUzdp3gQE2DIP08= =pnKm -----END PGP SIGNATURE----- --Sig_u4LaQi=.EbdLlgeI3g._33m-- From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 18:34:39 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBC5D16A417 for ; Fri, 19 Oct 2007 18:34:38 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id 8451613C468 for ; Fri, 19 Oct 2007 18:34:37 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id y2so822466uge for ; Fri, 19 Oct 2007 11:34:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=wwrDTEYCHOnX8qm7oXBj/7RAD9zB3DYmuzZNVo1DG6w=; b=SJwxcSbUldFfyavGAOOIiU/5Fw5GzL1LWgjRLOh+gooGDT7hO1SPs0Ue788TG1LNsx2iGQVMnCFoNVi/phuvcfjcQ2ZkZfH7QJMhqN2e1GiTum1tZqy18iSIK3cbOedOV7cijJmWv8d7ydzKPa8qeGahsvDQ5TojTfs6s6SR+Aw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=eYmcAMbhNPpAHCW/cr5V+gqqqmNnhqR/e3MkI7LYu06n1gLuko/8mh2+w9fw1EYolKfsQo0zysGYjXdOjgsJh1KB8m/HSy6JgLxMlBqZDDx4kRw7LWt2tNMezdC18qcJ+AmfFUpb5L8fKBaZlfmqTQi7WercmLVbkPIp6qzFew8= Received: by 10.78.170.6 with SMTP id s6mr1463311hue.1192813707906; Fri, 19 Oct 2007 10:08:27 -0700 (PDT) Received: by 10.78.97.18 with HTTP; Fri, 19 Oct 2007 10:08:27 -0700 (PDT) Message-ID: <3bbf2fe10710191008r6dc1939em85f8574e107af48b@mail.gmail.com> Date: Fri, 19 Oct 2007 19:08:27 +0200 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "John Baldwin" In-Reply-To: <200710190842.34286.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20071016094118.GE5411@hoeg.nl> <200710170916.18788.jhb@freebsd.org> <20071019045654.GF5411@hoeg.nl> <200710190842.34286.jhb@freebsd.org> X-Google-Sender-Auth: 5e7273eb9a32802a Cc: FreeBSD Hackers , Ed Schouten Subject: Re: Inner workings of turnstiles and sleepqueues X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 18:34:40 -0000 2007/10/19, John Baldwin : > On Friday 19 October 2007 12:56:54 am Ed Schouten wrote: > > * John Baldwin wrote: > > > The best option right now is to read the code. There are some comments in > > > both the headers and implementation. > > > > Would it be useful to write manpages for these interfaces, or do we > > assume that only godlike people can use them anyway? I am willing to > > write manpages for them. > > They already exist, but they really are only used to implement higher-level > primitives like locks and condition variables. The rest of the kernel should > use the higher-level primitives anyway. Well, really turnstiles don't have manpages, but this is still OK as they are only used in mutex while the "real" sleeping primitive should be identified by sleepqueues. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-hackers@FreeBSD.ORG Fri Oct 19 20:26:03 2007 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE89716A420 for ; Fri, 19 Oct 2007 20:26:03 +0000 (UTC) (envelope-from rermilov@team.vega.ru) Received: from mail.vega.ru (mx1.vega.ru [87.242.77.163]) by mx1.freebsd.org (Postfix) with ESMTP id 8289613C465 for ; Fri, 19 Oct 2007 20:25:57 +0000 (UTC) (envelope-from rermilov@team.vega.ru) Received: from [87.242.97.68] (port=63240 helo=edoofus.dev.vega.ru) by mail.vega.ru with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1Iixsy-0005XP-QD; Fri, 19 Oct 2007 19:51:56 +0000 Received: from edoofus.dev.vega.ru (localhost [127.0.0.1]) by edoofus.dev.vega.ru (8.14.1/8.14.1) with ESMTP id l9JJp7hl081170; Fri, 19 Oct 2007 23:51:07 +0400 (MSD) (envelope-from rermilov@team.vega.ru) Received: (from ru@localhost) by edoofus.dev.vega.ru (8.14.1/8.14.1/Submit) id l9JJp7ar081169; Fri, 19 Oct 2007 23:51:07 +0400 (MSD) (envelope-from rermilov@team.vega.ru) X-Authentication-Warning: edoofus.dev.vega.ru: ru set sender to rermilov@team.vega.ru using -f Date: Fri, 19 Oct 2007 23:51:07 +0400 From: Ruslan Ermilov To: Guy Helmer Message-ID: <20071019195107.GB40442@team.vega.ru> References: <4718C43B.2040504@palisadesys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4718C43B.2040504@palisadesys.com> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-hackers@freebsd.org Subject: Re: Netgraph ng_bridge status message missing last three values? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2007 20:26:04 -0000 On Fri, Oct 19, 2007 at 09:50:35AM -0500, Guy Helmer wrote: > I'm diagnosing a system that uses the Netgraph ng_bridge node to bridge two > Ethernet segments and the system seems to stop passing traffic at > randomly-triggered points (somewhat annoying for a system half a world away > behind a very restrictive firewall :-) > > I've installed a cron job to dump the Netgraph bridge status messages to a > log file every few minutes. However, I've found that the ng_bridge status > message looks like this: > > > ngctl msg bnet0: getstats 0 > STATS=Rec'd response "getstats" (4) from "[10]:": > Args: { recvOctets=474229151 recvPackets=3322699 recvMulticast=128640 > recvBroadcast=105813 recvUnknown=2372 xmitOctets=516954216 > xmitPackets=2968420 xmitMulticasts=8 xmitBroadcasts=339 } > > but to aid diagnosis I would really like to see several values that, > according to ng_bridge.h, are in the stats structure, including > recvRunts, recvInvalid, loopDrops, loopDetects, and memoryFailures. It > looks like all the code is there to report the values of these variables, > but I can't seem to find anything that is dropping these variables from > ngctl's output. > Struct members with zero values aren't shown. Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer