From owner-freebsd-current@FreeBSD.ORG Fri Nov 21 10:58:11 2008 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C9A31065672 for ; Fri, 21 Nov 2008 10:58:11 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from conversation.bsdunix.ch (ns1.bsdunix.ch [82.220.1.90]) by mx1.freebsd.org (Postfix) with ESMTP id BED198FC0C for ; Fri, 21 Nov 2008 10:58:10 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from localhost (localhost [127.0.0.1]) by conversation.bsdunix.ch (Postfix) with ESMTP id B08335D58; Fri, 21 Nov 2008 11:58:09 +0100 (CET) X-Virus-Scanned: by amavisd-new at mail.bsdunix.ch Received: from conversation.bsdunix.ch ([127.0.0.1]) by localhost (conversation.bsdunix.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id yf87IfgDmsph; Fri, 21 Nov 2008 11:58:06 +0100 (CET) Received: from 192.168.1.101.local.home (home.bsdunix.ch [82.220.17.23]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by conversation.bsdunix.ch (Postfix) with ESMTP id 1CDB55D56; Fri, 21 Nov 2008 11:58:06 +0100 (CET) Message-Id: From: Thomas Vogt To: Nikolay Denev In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Date: Fri, 21 Nov 2008 11:58:05 +0100 References: X-Mailer: Apple Mail (2.929.2) Cc: current@freebsd.org Subject: Re: deadlock with zfs? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 10:58:11 -0000 Thomas Vogt Am 21.11.2008 um 09:39 schrieb Nikolay Denev: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote: > >> Hello >> >> I encounter a deadlock while running a few rsync processes >> mirroring remote data to my local zfs pool. After a few hours my >> system is starting more and more vsftpd sessions without closing >> any inactive ftp sessions. I can't kill any rsync and vsftpd >> processes with "kill -9". Even shutdown -r now does not work. >> >> I got a few hunderts vsftpd processes like this >> >> 61346 root 1 57 0 7880K 1692K zfs 1 0:00 >> 0.00% vsftpd >> 61481 root 1 68 0 7880K 1696K zfs 1 0:00 >> 0.00% vsftpd >> 61354 root 1 65 0 7880K 1692K zfs 1 0:00 >> 0.00% vsftpd >> 61480 root 1 68 0 7880K 1696K zfs 0 0:00 >> 0.00% vsftpd >> 61600 root 1 69 0 7880K 1704K zfs 1 0:00 >> 0.00% vsftpd >> 61599 root 1 68 0 7880K 1704K zfs 1 0:00 >> 0.00% vsftpd >> >> Right now i'm building a debug kernel. Whats the best way to get >> usefull information from this deadlock? The system itself is not >> crashing and response well to ssh and i also have a serial console. >> >> The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008 >> (64bit) with zpool version 13. I use zfs only as data pool. The >> base system is running on ufs2: >> >> Filesystem Size Used Avail Capacity Mounted on >> /dev/da0s1a 496M 220M 236M 48% / >> devfs 1.0K 1.0K 0B 100% /dev >> /dev/da0s1g 169G 15G 141G 9% /disk1 >> /dev/da0s1f 3.9G 17M 3.5G 0% /tmp >> /dev/da0s1e 29G 3.7G 23G 14% /usr >> /dev/da0s1d 19G 7.1G 11G 40% /var >> pool 853G 0B 853G 0% /usr/local/data >> pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup >> pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp >> pool/portsnap 853G 633M 853G 0% /usr/local/data/ >> portsnap >> pool/www 853G 90M 853G 0% /usr/local/data/www >> >> loader.conf: >> vm.kmem_size="1G" >> kern.maxfiles="65536" >> kern.maxproc="20480" >> net.inet.tcp.tcbhashsize="4096" >> net.inet.tcp.hostcache.hashsize="1024" >> vfs.zfs.arc_min="64M" >> vfs.zfs.arc_max="768M" >> vfs.zfs.prefetch_disable="1" >> >> I also tried to set vfs.zfs.zil_disable=1 but the problem still >> exist. >> >> I know there are few deadlock reports listed in the freebsd wiki. >> Maybe we can trigger the root cause of the problem and fix it :) >> >> Regards >> Thomas >> > > Hi Thomas, > > from my recent experience it seems that tuning vm.kmem_size is no > longer required, > maybe you can try without it. Sure, i rebooted the system without vm.kmem_size. It takes a a few hours until i got deadlocks. Regards Thomas