From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 09:47:43 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 593C516A41B; Mon, 15 Oct 2007 09:47:43 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id EB1AA13C44B; Mon, 15 Oct 2007 09:47:42 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 1D63845E93; Mon, 15 Oct 2007 11:47:41 +0200 (CEST) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id CE09045683; Mon, 15 Oct 2007 11:47:28 +0200 (CEST) Date: Mon, 15 Oct 2007 11:47:16 +0200 From: Pawel Jakub Dawidek To: Claus Guttesen , freebsd-fs@freebsd.org, freebsd-current@freebsd.org Message-ID: <20071015094716.GG10170@garage.freebsd.pl> References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071015082629.GB1290@tirith.brixandersen.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="kadn00tgSopKmJ1H" Content-Disposition: inline In-Reply-To: <20071015082629.GB1290@tirith.brixandersen.dk> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2007 09:47:43 -0000 --kadn00tgSopKmJ1H Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 15, 2007 at 10:26:29AM +0200, Henrik Brix Andersen wrote: > On Mon, Oct 08, 2007 at 07:18:10PM +0200, Claus Guttesen wrote: > > > I was able to reproduce the panic by rsyncing big files and trying > > > bonnie++ test suggested in this thread. > > > > > > Can you guys retry with this patch: > > > > > > http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch > > > > > > It's a hack, yes, but allows to mitigate the problem quite well. I'm > > > looking for a solution that can be used for 7.0 before we find a bett= er > > > fix. > >=20 > > Congrats Pawel! You made my server survive my rsync of 90 GB. :-) > >=20 > > This is on same src as the one that required a reboot except for your > > patch. So this fix does 'alleviate kmem_map too small' in my case. >=20 > While we have come across the 'kmem_map too small' panics in the past, > these are now solved - but our rsync processes still hang at random, > just spinning the CPU. This is when running multiple rsyncs to the > same zpool: >=20 > http://www.brixandersen.dk/tmp/zfs-rsync-debug.1.txt This looks like VFS deadlock between process 42575 and 42585. You debugging is really nice, but unfortunately there is no backtrace of process 42575, so we don't know where and why it's stuck... Can you next time take 'show lockedvnods', 'show alllocks' (as you did this time), but also 'alltrace'? --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --kadn00tgSopKmJ1H Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHEzckForvXbEpPzQRAv5hAJ4jSUxA01dxxvRcFHaAbalKQzGJ9wCgtaBV 2r3jy+/H/uoQlWWzbnTlrwA= =ywo+ -----END PGP SIGNATURE----- --kadn00tgSopKmJ1H--