From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 12:26:40 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 713DA16A41A for ; Fri, 1 Feb 2008 12:26:40 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id CFF0C13C467 for ; Fri, 1 Feb 2008 12:26:39 +0000 (UTC) (envelope-from prvs=1917699425=killing@multiplay.co.uk) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1201868606; x=1202473406; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=gYmfS3sPuwK7qiFAS1wEi 83iwqAJQZF6GSfVvDwFWlw=; b=HYDx2FtsPN4OHhDG6POGIXPr5QwYONgZkEt+n jDb1dxBOttyPUHq5JxQ0D7yvKg6UAgZf9swUupxxWhA914r1f2ifVRAgVmxb9Q4a CBdgplWMciLH0w1VxeCfI8sugCwWMCfwzsjEFzotn9PycWeKGZ5F332btinXn8vk 7H9fvk= X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-14.7 required=6.0 tests=BAYES_00, USER_IN_WHITELIST, USER_IN_WHITELIST_TO autolearn=ham version=3.1.8 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v9.6.3) with ESMTP id md50004973364.msg; Fri, 01 Feb 2008 12:23:24 +0000 Message-ID: <012401c864cd$3c0c9530$b6db87d4@multiplay.co.uk> From: "Steven Hartland" To: , "Ivan Voras" References: <200801310147.BAA04522@sopwith.solgatos.com> <47A2A606.9080702@freebsd.org><002201c86499$7861ac20$b6db87d4@multiplay.co.uk> Date: Fri, 1 Feb 2008 12:23:21 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 212.135.219.182 X-Return-Path: prvs=1917699425=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-Spam-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 12:23:25 +0000 X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 01 Feb 2008 12:23:26 +0000 Cc: Subject: Re: newfs locks entire machine for 20seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 12:26:40 -0000 ----- Original Message ----- From: "Ivan Voras" ... >> geom debugging I get:- >> Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, >> 0xffffff00010e6100, 2, 0) >> Feb 1 06:04:45 geomtest kernel: ref 0xffffff00010e6100 >> Feb 1 06:04:45 geomtest kernel: g_post_event_x(0xffffffff802394c0, >> 0xffffff00014e6700, 2, 0) >> Feb 1 06:04:45 geomtest kernel: ref 0xffffff00014e6700 >> Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, >> 0xffffff00010e6100, 2, 0) >> Feb 1 06:04:49 geomtest kernel: ref 0xffffff00010e6100 >> Feb 1 06:04:49 geomtest kernel: g_post_event_x(0xffffffff80239270, >> 0xffffff00014e6700, 2, 0) >> Feb 1 06:04:49 geomtest kernel: ref 0xffffff00014e6700 >> Feb 1 06:04:49 geomtest kernel: mbr_taste(MBR,da0s3) >> Feb 1 06:04:49 geomtest kernel: g_mbrext_taste(MBREXT,da0s3) >> Feb 1 06:04:49 geomtest kernel: g_slice_spoiled(0xffffff0001b27180/da0s3) >> Feb 1 06:04:49 geomtest kernel: g_wither_geom(0xffffff0001a33c00(da0s3)) >> Feb 1 06:04:49 geomtest kernel: g_part_taste(PART,da0s3) >> Feb 1 06:04:56 geomtest kernel: g_post_event_x(0xffffffff80235b10, >> 0xffffff000144a9c0, 2, 262144) >> Feb 1 06:05:00 geomtest kernel: g_wither_geom(0xffffff000158b300(da0s3)) >> Feb 1 06:05:00 geomtest kernel: >> Feb 1 06:05:00 geomtest kernel: g_label_taste(LABEL, da0s3) >> Feb 1 06:05:00 geomtest kernel: >> Feb 1 06:05:16 geomtest kernel: GEOM_LABEL[1]: MSDOSFS: da0s3: FAT32 >> volume not valid. >> Feb 1 06:05:16 geomtest kernel: g_detach(0xffffff0001b23980) >> Feb 1 06:05:16 geomtest kernel: g_destroy_consumer(0xffffff0001b23980) >> Feb 1 06:05:16 geomtest kernel: > >> So after all that I can see why the sysctl call is taking >> so long to complete but the burning question is why does > > Can you explain - I don't see it :) Do you mean to say there's a > contention for sysctl lock between geom_confxml and g_waitfor_event or > that geom_label tasting has something to do with it? Nope what I belive to be happening is this sysctl_kern_geom_confxml which is called while SYSCTL_LOCK is held from userland_sysctl returns EAGAIN for a considerable period while newfs runs. >> the entire system lock because of this? What else is >> waiting on the sysctl lock which is so critical? > > What I do know is that sysctl is GIANT-locked, which is also used by > some parts of device handling infrastructure (dead_cdevsw), the USB > stack, and can creep itself in the timer via swi_sched in > subr_taskqueue.c:303. I cannot say for sure that's what happening here, > but they are possibilities. > > If you can provoke this reliably, I think there is a (old!) patch for > removing sysctl from under the Giant lock that you could try. Yes this is 100% reproducable and I'm beginning to suspect this issue may the cause of the regular pauses on our hosting machines so it might have further reaching effects than just this one example. If SYSCTL_LOCK is indeed GIANT locked then it might explain why its effect is so pronounced. Tried ADAPTIVE_SX as test to see if that helped as the sysctl lock is and sx but didnt help Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.