From owner-freebsd-fs@FreeBSD.ORG Mon Aug 24 12:57:40 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72295106564A for ; Mon, 24 Aug 2009 12:57:40 +0000 (UTC) (envelope-from p.christias@noc.ntua.gr) Received: from achilles.noc.ntua.gr (achilles.noc.ntua.gr [IPv6:2001:648:2000:de::210]) by mx1.freebsd.org (Postfix) with ESMTP id C88398FC08 for ; Mon, 24 Aug 2009 12:57:39 +0000 (UTC) Received: from ajax.noc.ntua.gr (ajax6.noc.ntua.gr [IPv6:2001:648:2000:dc::1]) by achilles.noc.ntua.gr (8.14.3/8.14.3) with ESMTP id n7OCvbjJ027060 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 24 Aug 2009 15:57:37 +0300 (EEST) (envelope-from p.christias@noc.ntua.gr) Received: from ajax.noc.ntua.gr (localhost [127.0.0.1]) by ajax.noc.ntua.gr (8.14.3/8.14.3) with ESMTP id n7OCvbge094090 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 24 Aug 2009 15:57:37 +0300 (EEST) (envelope-from p.christias@noc.ntua.gr) Received: (from christia@localhost) by ajax.noc.ntua.gr (8.14.3/8.14.3/Submit) id n7OCvbrC094089; Mon, 24 Aug 2009 15:57:37 +0300 (EEST) (envelope-from p.christias@noc.ntua.gr) X-Authentication-Warning: ajax.noc.ntua.gr: christia set sender to p.christias@noc.ntua.gr using -f Date: Mon, 24 Aug 2009 15:57:37 +0300 From: Panagiotis Christias To: "Mikhail (Plus Plus)" Message-ID: <20090824125737.GA92643@noc.ntua.gr> References: <4A927CB3.3040402@plus-plus.su> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4A927CB3.3040402@plus-plus.su> User-Agent: Mutt/1.5.18 (2008-05-17) X-Virus-Scanned: ClamAV version 0.94.2, clamav-milter version 0.94.2 on achilles.noc.ntua.gr X-Virus-Scanned: clamav-milter 0.95.2 at ajax.noc.ntua.gr X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: Re: need help with ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2009 12:57:40 -0000 On Mon, Aug 24, 2009 at 03:42:43PM +0400, Mikhail (Plus Plus) wrote: > Greetings gentlemen, > > I need your help with ZFS -- how I can diagnose, debug ZFS crashes, and > if possible make it more stable. We're running raidz pool and we're > having hard time to get it running smooth -- ZFS simply crashes as soon > as we put some load on it. > > Couple months back we've built custom server to replace our old storage > system. New server has 7 x 1GB SATA drives, Intel Q6600 Quad core CPU, > 8GB RAM. OS: FreeBSD 7.2-RELEASE-p2 amd64, stock GENERIC kernel. > > So I decided to try ZFS, and I was amazed by it's features. I've read > ZFS wiki page and my loader.conf follows: > > zen# cat loader.conf > vm.kmem_size="1536M" > vm.kmem_size_max="3072M" > vm.pmap.shpgperproc="1024" > vfs.zfs.arc_min="256M" > vfs.zfs.arc_max="384M" > vfs.zfs.vdev.cache.size="50M" > vfs.zfs.prefetch_disable="1" > kern.maxproc="20000" > zen# > > and > > zen# zpool status > pool: datapool > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > datapool ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > ad24 ONLINE 0 0 0 > ad8 ONLINE 0 0 0 > ad18 ONLINE 0 0 0 > ad20 ONLINE 0 0 0 > ad22 ONLINE 0 0 0 > ad10 ONLINE 0 0 0 > spares > ad26 AVAIL > > errors: No known data errors > zen# zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > datapool 5.44T 3.54T 1.90T 65% ONLINE - > zen# > > > Problem starts as soon as we put some load on FS - e.g. last night I > tried to rsync 1TB of different files from above ZFS pool to single hard > drive (partitioned UFS2), but server crashed and restarted after > copying ~700GB of data. I also tried to stress-test it by running 100 > torrent downloads (using rtorrent), and server also crashes and restarts > after running for about 30-40 minutes. > My guess these crashes happen due to big load on filesystem - ZFS eats > all available memory and then server simply crashes. > > Right now I'm completely lost - I can't even copy 1TB from ZFS to > another partition.. > How can I diagnose the issue? Is there anything available to make ZFS > more stable? I would suggest you try FreeBSD 8.0 that includes the latest version of ZFS (version 13), which fixed several problems present in 7.x. Then, check for crash dumps (see dumpon(8)), collect any available info and sent it to the list. Regards, Panagiotis -- Panagiotis J. Christias Network Management Center P.Christias@noc.ntua.gr National Technical Univ. of Athens, GREECE