From owner-freebsd-current@FreeBSD.ORG Fri Nov 21 09:12:27 2008 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7867106564A for ; Fri, 21 Nov 2008 09:12:27 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by mx1.freebsd.org (Postfix) with ESMTP id 3ED218FC14 for ; Fri, 21 Nov 2008 09:12:27 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so611177fgb.35 for ; Fri, 21 Nov 2008 01:12:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:cc:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-pgp-agent:x-mailer; bh=1fJD2rJSnyKLO4WmpWf/CS6X9Iy9i4i+0gu33Y+8r8A=; b=DHphQ4w++6quDfJ4yt6OIR5nHxYy/1WqBYQ3zHxedj1fEDpT5CRpD0xWK3DbGQb2Tj 9vImyOZ36k/5MSHiKH+G600cIP/CCZosjxCNuJFarpGoANC1HbDhUyPdUHndQimsTgHK CcLO/OegEcdmDj1TTsRmoncHhnUEtPNszd1bM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=cc:message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-pgp-agent:x-mailer; b=pm/JvlgTkwWVapEP51XqOpyh/CdvfbWr3QjadsLyRqSfwL11r1HswN9Lb6NjS8vMxL kN144Tsufr1+loSXnXcSdqmXQ5NSjaR37adSTw7MUOD/mbhK526HEIV2eGLGEUVIR0dk WOlFeBwlNAO0RSxuBbeW07nbP6D1TsVx376dA= Received: by 10.181.144.10 with SMTP id w10mr100483bkn.30.1227256803352; Fri, 21 Nov 2008 00:40:03 -0800 (PST) Received: from ndenev.cmotd.com (blah.sun-fish.com [217.18.249.150]) by mx.google.com with ESMTPS id p9sm2762118fkb.5.2008.11.21.00.39.59 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 21 Nov 2008 00:40:00 -0800 (PST) Message-Id: From: Nikolay Denev To: Thomas Vogt In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Fri, 21 Nov 2008 10:39:58 +0200 References: X-Pgp-Agent: GPGMail d53 (v53, Leopard) X-Mailer: Apple Mail (2.929.2) Cc: current@freebsd.org Subject: Re: deadlock with zfs? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 09:12:27 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote: > Hello > > I encounter a deadlock while running a few rsync processes mirroring > remote data to my local zfs pool. After a few hours my system is > starting more and more vsftpd sessions without closing any inactive > ftp sessions. I can't kill any rsync and vsftpd processes with "kill > -9". Even shutdown -r now does not work. > > I got a few hunderts vsftpd processes like this > > 61346 root 1 57 0 7880K 1692K zfs 1 0:00 0.00% > vsftpd > 61481 root 1 68 0 7880K 1696K zfs 1 0:00 0.00% > vsftpd > 61354 root 1 65 0 7880K 1692K zfs 1 0:00 0.00% > vsftpd > 61480 root 1 68 0 7880K 1696K zfs 0 0:00 0.00% > vsftpd > 61600 root 1 69 0 7880K 1704K zfs 1 0:00 0.00% > vsftpd > 61599 root 1 68 0 7880K 1704K zfs 1 0:00 0.00% > vsftpd > > Right now i'm building a debug kernel. Whats the best way to get > usefull information from this deadlock? The system itself is not > crashing and response well to ssh and i also have a serial console. > > The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008 > (64bit) with zpool version 13. I use zfs only as data pool. The base > system is running on ufs2: > > Filesystem Size Used Avail Capacity Mounted on > /dev/da0s1a 496M 220M 236M 48% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/da0s1g 169G 15G 141G 9% /disk1 > /dev/da0s1f 3.9G 17M 3.5G 0% /tmp > /dev/da0s1e 29G 3.7G 23G 14% /usr > /dev/da0s1d 19G 7.1G 11G 40% /var > pool 853G 0B 853G 0% /usr/local/data > pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup > pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp > pool/portsnap 853G 633M 853G 0% /usr/local/data/ > portsnap > pool/www 853G 90M 853G 0% /usr/local/data/www > > loader.conf: > vm.kmem_size="1G" > kern.maxfiles="65536" > kern.maxproc="20480" > net.inet.tcp.tcbhashsize="4096" > net.inet.tcp.hostcache.hashsize="1024" > vfs.zfs.arc_min="64M" > vfs.zfs.arc_max="768M" > vfs.zfs.prefetch_disable="1" > > I also tried to set vfs.zfs.zil_disable=1 but the problem still exist. > > I know there are few deadlock reports listed in the freebsd wiki. > Maybe we can trigger the root cause of the problem and fix it :) > > Regards > Thomas > Hi Thomas, from my recent experience it seems that tuning vm.kmem_size is no longer required, maybe you can try without it. - -- Regards, Nikolay Denev -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (Darwin) iEYEARECAAYFAkkmc94ACgkQHNAJ/fLbfrlQNwCfYOgJoxqPMUeQ0JNsuES1Gd7a T8QAnjf/bAj3LIUDjZlRIoFHkNl9l62q =hVLJ -----END PGP SIGNATURE-----