Date: Fri, 21 Nov 2008 15:52:47 +0100 From: Thomas Vogt <freebsdlists@bsdunix.ch> To: Nikolay Denev <ndenev@gmail.com> Cc: current@freebsd.org Subject: Re: deadlock with zfs? Message-ID: <4926CB3F.8010500@bsdunix.ch> In-Reply-To: <C4034F1E-9B32-4C45-881D-664456714345@bsdunix.ch> References: <F1687236-D85B-467F-9884-AABE9EA64C6C@bsdunix.ch> <B647D506-25CB-4314-BB1B-E9AD5044C92C@gmail.com> <C4034F1E-9B32-4C45-881D-664456714345@bsdunix.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello > Am 21.11.2008 um 09:39 schrieb Nikolay Denev: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> >> On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote: >> >>> Hello >>> >>> I encounter a deadlock while running a few rsync processes mirroring >>> remote data to my local zfs pool. After a few hours my system is >>> starting more and more vsftpd sessions without closing any inactive >>> ftp sessions. I can't kill any rsync and vsftpd processes with "kill >>> -9". Even shutdown -r now does not work. >>> >>> I got a few hunderts vsftpd processes like this >>> >>> 61346 root 1 57 0 7880K 1692K zfs 1 0:00 0.00% >>> vsftpd >>> 61481 root 1 68 0 7880K 1696K zfs 1 0:00 0.00% >>> vsftpd >>> 61354 root 1 65 0 7880K 1692K zfs 1 0:00 0.00% >>> vsftpd >>> 61480 root 1 68 0 7880K 1696K zfs 0 0:00 0.00% >>> vsftpd >>> 61600 root 1 69 0 7880K 1704K zfs 1 0:00 0.00% >>> vsftpd >>> 61599 root 1 68 0 7880K 1704K zfs 1 0:00 0.00% >>> vsftpd >>> >>> Right now i'm building a debug kernel. Whats the best way to get >>> usefull information from this deadlock? The system itself is not >>> crashing and response well to ssh and i also have a serial console. >>> >>> The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008 >>> (64bit) with zpool version 13. I use zfs only as data pool. The base >>> system is running on ufs2: >>> >>> Filesystem Size Used Avail Capacity Mounted on >>> /dev/da0s1a 496M 220M 236M 48% / >>> devfs 1.0K 1.0K 0B 100% /dev >>> /dev/da0s1g 169G 15G 141G 9% /disk1 >>> /dev/da0s1f 3.9G 17M 3.5G 0% /tmp >>> /dev/da0s1e 29G 3.7G 23G 14% /usr >>> /dev/da0s1d 19G 7.1G 11G 40% /var >>> pool 853G 0B 853G 0% /usr/local/data >>> pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup >>> pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp >>> pool/portsnap 853G 633M 853G 0% /usr/local/data/portsnap >>> pool/www 853G 90M 853G 0% /usr/local/data/www >>> >>> loader.conf: >>> vm.kmem_size="1G" >>> kern.maxfiles="65536" >>> kern.maxproc="20480" >>> net.inet.tcp.tcbhashsize="4096" >>> net.inet.tcp.hostcache.hashsize="1024" >>> vfs.zfs.arc_min="64M" >>> vfs.zfs.arc_max="768M" >>> vfs.zfs.prefetch_disable="1" >>> >>> I also tried to set vfs.zfs.zil_disable=1 but the problem still exist. >>> >>> I know there are few deadlock reports listed in the freebsd wiki. >>> Maybe we can trigger the root cause of the problem and fix it :) >>> >>> Regards >>> Thomas >>> >> >> Hi Thomas, >> >> from my recent experience it seems that tuning vm.kmem_size is no >> longer required, >> maybe you can try without it. > > Sure, i rebooted the system without vm.kmem_size. It takes a a few hours > until i got deadlocks. Removing vm.kmem_size didn't fix my deadlock issue. It just took a bit longer until the deadlock was triggered. Maybe someone can check my deadlock debug output at http://www.bsdunix.ch/deadlock_2008-11-21.txt Regards, Thomas
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4926CB3F.8010500>