From owner-freebsd-current@FreeBSD.ORG Mon Sep 20 14:01:31 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from alona.my.domain (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 86B3E10656A8; Mon, 20 Sep 2010 14:01:29 +0000 (UTC) (envelope-from davidxu@freebsd.org) Message-ID: <4C976934.5040501@freebsd.org> Date: Mon, 20 Sep 2010 22:01:24 +0800 From: David Xu User-Agent: Thunderbird 2.0.0.21 (X11/20090522) MIME-Version: 1.0 To: Lawrence Stewart References: <4C961009.6080609@freebsd.org> <4C96ED40.7070206@DataIX.net> <4C976F14.8000408@freebsd.org> <4C9789FD.5000008@freebsd.org> <4C9754CD.7000508@freebsd.org> In-Reply-To: <4C9754CD.7000508@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org Subject: Re: sysctl -a is slow X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Sep 2010 14:01:31 -0000 Lawrence Stewart wrote: > On 09/21/10 02:21, David Xu wrote: > >> jhell wrote: >> >>> On Mon, 20 Sep 2010 10:26, David Xu wrote: >>> In Message-Id: <4C976F14.8000408@freebsd.org> >>> >>> >>>> jhell wrote: >>>> >>>>> On 09/19/2010 09:28, David Xu wrote: >>>>> >>>>>> just typed sysctl -a on keyboard, and found it is slow, sometimes >>>>>> it has been stuck for a few seconds, further studied,I found it is >>>>>> stucked at sysctl kern.geom: >>>>>> >>>>>> %/usr/bin/time sysctl -a kern.geom >>>>>> kern.geom.collectstats: 1 >>>>>> kern.geom.debugflags: 0 >>>>>> kern.geom.label.debug: 0 >>>>>> kern.geom.label.ext2fs.enable: 1 >>>>>> kern.geom.label.iso9660.enable: 1 >>>>>> kern.geom.label.msdosfs.enable: 1 >>>>>> kern.geom.label.ntfs.enable: 1 >>>>>> kern.geom.label.reiserfs.enable: 1 >>>>>> kern.geom.label.ufs.enable: 1 >>>>>> kern.geom.label.ufsid.enable: 1 >>>>>> kern.geom.label.gptid.enable: 1 >>>>>> kern.geom.label.gpt.enable: 1 >>>>>> 2.01 real 0.00 user 0.00 sys >>>>>> >>>>>> it seems it needs more than 2 seconds to complete. >>>>>> >>>>>> >>>>> A ktrace(1) and a kdump(1) of the resulting ktrace.out file would >>>>> probably help here along with uname -a. Ive seen this happen once >>>>> before >>>>> but do not recall what caused it. >>>>> >>>>> >>>>> Regards & good luck, >>>>> >>>>> >>>> Result is dumped here. >>>> http://people.freebsd.org/~davidxu/sysctl_slow.txt >>>> I think the culprit is sysctl kern.geom.confdot, >>>> which does not appear in normal output, until I check the kdump result. >>>> I tried five times, and it was blocked three times. >>>> >>>> >>> Inspecting the output of sysctl -b kern.geom.confdot will give you >>> what you currently have configured in the system as disks and what not >>> through geom. If this seems to be bailing at that point, which is an >>> opaque MIB/OID which doesn't come up other than when you use the "-o" >>> switch to sysctl(1) then could you check your labels for your disks >>> for any weird characters in the labels ? >>> >>> ( sysctl -bo kern.geom ) >>> >>> Also does this have the same effect when run in a xterm, cons25 >>> terminal ? >>> >>> And same for the above but with the C, *_COUNTRY.UTF-8 or your normal >>> locale ? >>> >>> ( env LANG=C sysctl kern.geom ) >>> >>> Looking at the output from mine there are quite a few unprintable >>> characters present. Maybe these are having an impact with one of your >>> labels. >>> >>> >>> >> I redirect all output to a disk file, and it still needs 1 second to >> complete, this machine is dual-core pentium E5500, faster than previous >> one which is a dual-core AMD 5000+ machine, the 5000+ needs 2 >> seconds to complete. >> >> $/usr/bin/time sysctl -b kern.geom.confdot > sysctl_geom_confdot.txt >> 1.00 real 0.00 user 0.00 sys >> >> the file is here: >> http://people.freebsd.org/~davidxu/sysctl_geom_confdot.txt >> > > As an extra data point, running "/usr/bin/time sysctl -b > kern.geom.confdot" repeatedly on my amd64 8.1-STABLE desktop varies > between 0s and 2s. It reports 0 majority of the time but every 5 or so > runs it'll stall for 1 or 2 seconds. So the problem isn't isolated to head. > > Cheers, > Lawrence > I happened to set kern.sched.preempt_thresh=200, so the kernel is more aggressive than default on thread preemption. it is easier than default to reproduce the problem, my desktop machine is idle, but it still stalls 1 or 2 seconds on the sysctl.