From owner-freebsd-stable@FreeBSD.ORG Thu Mar 7 06:55:44 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1767B5FA for ; Thu, 7 Mar 2013 06:55:44 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id C054F2A7 for ; Thu, 7 Mar 2013 06:55:43 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r276taoZ052460 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 6 Mar 2013 22:55:36 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r276ta7w052459; Wed, 6 Mar 2013 22:55:36 -0800 (PST) (envelope-from jmg) Date: Wed, 6 Mar 2013 22:55:36 -0800 From: John-Mark Gurney To: Karl Denninger Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults? Message-ID: <20130307065536.GA50035@funkthat.com> Mail-Followup-To: Karl Denninger , freebsd-stable@freebsd.org References: <513524B2.6020600@denninger.net> <89680320E0FA4C0A99D522EA2037CE6E@multiplay.co.uk> <20130305050539.GA52821@anubis.morrow.me.uk> <20130305053249.GA38107@icarus.home.lan> <545CD2ABE3D146F2B91963ADF6090CDE@multiplay.co.uk> <20130305092700.GA43045@icarus.home.lan> <5135EB62.6060006@denninger.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5135EB62.6060006@denninger.net> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 06 Mar 2013 22:55:36 -0800 (PST) Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2013 06:55:44 -0000 Karl Denninger wrote this message on Tue, Mar 05, 2013 at 06:56 -0600: > When it happens on my system anything that is CPU-bound continues to > execute. I can switch consoles and network I/O also works. If I have > an iostat running at the time all I/O counters go to and remain at zero > while the stall is occurring, but the process that is producing the > iostat continues to run and emit characters whether it is a ssh session > or on the physical console. > > The CPUs are running and processing, but all threads block if they > attempt access to the disk I/O subsystem, irrespective of the portion of > the disk I/O subsystem they attempt to access (e.g. UFS, swap or ZFS) I > therefore cannot start any new process that requires image activation. Since it seems like there is a thread that is spinning... Has anyone thought to modify kgdb to mlockall it's memory and run it against the current system (kgdb /boot/kernel/kernel /dev/mem), and then when the thread goes busy, use kgdb to see what where it's spinning? Just a thought... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."