From owner-freebsd-current@FreeBSD.ORG Wed Jun 20 17:03:12 2007 Return-Path: X-Original-To: Current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3356D16A421 for ; Wed, 20 Jun 2007 17:03:12 +0000 (UTC) (envelope-from marcus@FreeBSD.org) Received: from av-tac-rtp.cisco.com (hen.cisco.com [64.102.19.198]) by mx1.freebsd.org (Postfix) with ESMTP id BC70513C484 for ; Wed, 20 Jun 2007 17:03:11 +0000 (UTC) (envelope-from marcus@FreeBSD.org) X-TACSUNS: Virus Scanned Received: from rooster.cisco.com (localhost [127.0.0.1]) by av-tac-rtp.cisco.com (8.11.7p3+Sun/8.11.7) with ESMTP id l5KH35I01446; Wed, 20 Jun 2007 13:03:05 -0400 (EDT) Received: from [64.102.193.121] (dhcp-64-102-193-121.cisco.com [64.102.193.121]) by rooster.cisco.com (8.11.7p3+Sun/8.11.7) with ESMTP id l5KH2xq29415; Wed, 20 Jun 2007 13:03:04 -0400 (EDT) Message-ID: <46795DD2.6000505@FreeBSD.org> Date: Wed, 20 Jun 2007 13:03:14 -0400 From: Joe Marcus Clarke Organization: FreeBSD, Inc. User-Agent: Thunderbird 2.0.0.4 (Macintosh/20070604) MIME-Version: 1.0 To: Kris Kennaway References: <1182354823.6504.23.camel@shumai.marcuscom.com> <20070620160306.GA74674@rot26.obsecurity.org> <1182356274.6504.30.camel@shumai.marcuscom.com> <20070620162854.GA35000@rot26.obsecurity.org> In-Reply-To: <20070620162854.GA35000@rot26.obsecurity.org> X-Enigmail-Version: 0.95.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Current@FreeBSD.org Subject: Re: ZFS and deadlock with {nullfs,NFS} X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jun 2007 17:03:12 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Kris Kennaway wrote: > On Wed, Jun 20, 2007 at 12:17:54PM -0400, Joe Marcus Clarke wrote: >> On Wed, 2007-06-20 at 12:03 -0400, Kris Kennaway wrote: >>> On Wed, Jun 20, 2007 at 11:53:43AM -0400, Joe Marcus Clarke wrote: >>>> I've resurrected by amd64 Tinderbox with a ZFS base, and I've been >>>> seeing a 100% reproducible deadlock when I use it with either localhost >>>> NFS or nullfs. When this occurs, the CPU is 100% idle, but I can no >>>> longer connect via SSH, and the box will only reboot from the debugger. >>>> I know there are some tuning bits I can tweak, but all I've run across >>>> is for memory consumption. Any pointers would be helpful. I'm also at >>>> the debugger, so if there is anything I can do to help troubleshoot why >>>> this is happening, please let me know. >>>> >>>> This box is -CURRENT as of June 19, 2007. It has a GENERIC kernel minus >>>> devices I do not have (i.e. SMP kernel). I am currently using nullfs >>>> for the Tinderbox. The process that most regularly locks up is mtree. >>>> Here is the trace: >>>> A full process list from the debugger can be found at >>>> http://www.marcuscom.com/downloads/cobbler_proc.txt . >>> 404 at the moment, but look for processes involving zil* in the >>> backtrace. I had to disable zil (vfs.zfs.zil_disable=1 tunable) to >>> prevent low-memory deadlocks on my machines. Since then it's been >>> fine. >> Fixed, sorry. >> >>> You may also wish to use my patches (see the archives) to improve >>> performance and low-memory behaviour. >> Thanks for the advice. I'll check. I didn't think low memory since it >> didn't look like I was using much. Even now with the box locked, I have >> 1035 MB free with no swap in use (this box has 2 GB total). > > By default there is only a 320 MB kmem_map into which all of zfs > (including its buffer cache and I/O buffers) has to cram itself, so > that is where the low memory condition may be happening. This is one > of the things that should be tuned to give non-terrible performance by > actually allowing some caching to occur. Dammit! I always forget about KVM. Thanks. :-} Joe - -- Joe Marcus Clarke FreeBSD GNOME Team :: gnome@FreeBSD.org FreeNode / #freebsd-gnome http://www.FreeBSD.org/gnome -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGeV3Rb2iPiv4Uz4cRAifqAJwOzab1frEwY0iQLq/WFGORjD/mFQCeMa6R WpY8L1tg5Yx4Qw70NulLTp4= =XOcd -----END PGP SIGNATURE-----