From owner-freebsd-fs@FreeBSD.ORG Thu May 3 19:59:43 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9D6C616A400; Thu, 3 May 2007 19:59:43 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 7EA1213C457; Thu, 3 May 2007 19:59:43 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 47C3A5B2E; Thu, 3 May 2007 12:59:43 -0700 (PDT) To: Pawel Jakub Dawidek In-reply-to: Your message of "Fri, 27 Apr 2007 16:26:06 +0200." <20070427142606.GK49413@garage.freebsd.pl> Date: Thu, 03 May 2007 12:59:43 -0700 From: Bakul Shah Message-Id: <20070503195943.47C3A5B2E@mail.bitblocks.com> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: kmem_map too small panic again X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 May 2007 19:59:43 -0000 > On Fri, Apr 27, 2007 at 12:35:35AM +0200, Pawel Jakub Dawidek wrote: > > On Thu, Apr 26, 2007 at 02:33:47PM -0700, Bakul Shah wrote: > > > An update: > > > > > > I reverted sources to to Apr 24 16:49 UTC and rebuilt the > > > kernel and the bug goes away -- I was able to restore 53GB > > > (840K+ inodes) and do a bunch of du with no problems. > > > > > > But the bug remains on a kernel with the latest zfs changes. > > > All I have to do run du a couple of times in the restored > > > tree to crash the system. There is no crash with multiple du > > > on a similarly sized UFS2, only on ZFS. This is on a > > > Athlon64 X2 Dual Core Processor 3800+ running in 32 bit mode. > > > The exact message is: > > > > > > panic: kmem_malloc(98304): kmem_map too small: 335478784 total allocated > > > > I can reproduce it and I'm working on it. > > The problem is that kern.maxvnodes are tuned based on vnode+UFS_inode > size. In case of ZFS, the size of vnode+ZFS_znode_dnode+dmu_buf is > larger. As a work-around just decrease kern.maxvnodes to something like > 3/4 of the current value. Pawel, thank you for this fix; I have been running -current with it for a few days but as others have reported, this does not fix the problem, only makes it much less likely -- or may be there is another problem. At least now I can get a crash dump! I have two filesystems in one pool with about 1.28M inodes altogether. Based on a few trials it seems it is necessary to walk them both before triggering this panic (or may be it is a function of how many inodes are statted). Every second I sent output of vmstat -z to another machine during testing. Nothing pops out as obviously wrong but here are somethings worth looking at: $ grep ^512 vmstream | firstlast 512: 512, 0, 466, 38, 512: 512, 0, 131491, 3669, $ grep ' Slab' vmstream | firstlast UMA Slabs: 64, 0, 2516, 21, UMA Slabs: 64, 0, 23083, 222, $ grep dmu vmstream | firstlast dmu_buf_impl_t: 140, 0, 912, 68, dmu_buf_impl_t: 140, 0, 136034, 3938, $ grep znode vmstream | firstlast zfs_znode_cache: 236, 0, 261, 43, zfs_znode_cache: 236, 0, 64900, 5596, # firstlast displays the first and last line of its input.