From owner-freebsd-fs@FreeBSD.ORG Mon Aug 24 12:28:43 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0EA2C106568E for ; Mon, 24 Aug 2009 12:28:43 +0000 (UTC) (envelope-from m@plus-plus.su) Received: from ext-mail2.ux6.net (ext-mail2.ux6.net [213.163.72.53]) by mx1.freebsd.org (Postfix) with ESMTP id C86BA8FC19 for ; Mon, 24 Aug 2009 12:28:42 +0000 (UTC) Received: from ermik.ux6.net ([91.206.231.146]) by ext-mail2.ux6.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MfXva-0009Cm-L2 for freebsd-fs@freebsd.org; Mon, 24 Aug 2009 13:41:34 +0200 Received: from [206.161.120.61] (helo=[192.168.5.15]) by ermik.ux6.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1MfXvW-000ANr-0W for freebsd-fs@freebsd.org; Mon, 24 Aug 2009 15:41:30 +0400 Message-ID: <4A927CB3.3040402@plus-plus.su> Date: Mon, 24 Aug 2009 15:42:43 +0400 From: "Mikhail (Plus Plus)" User-Agent: Thunderbird 2.0.0.17 (X11/20080925) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: need help with ZFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2009 12:28:43 -0000 Greetings gentlemen, I need your help with ZFS -- how I can diagnose, debug ZFS crashes, and if possible make it more stable. We're running raidz pool and we're having hard time to get it running smooth -- ZFS simply crashes as soon as we put some load on it. Couple months back we've built custom server to replace our old storage system. New server has 7 x 1GB SATA drives, Intel Q6600 Quad core CPU, 8GB RAM. OS: FreeBSD 7.2-RELEASE-p2 amd64, stock GENERIC kernel. So I decided to try ZFS, and I was amazed by it's features. I've read ZFS wiki page and my loader.conf follows: zen# cat loader.conf vm.kmem_size="1536M" vm.kmem_size_max="3072M" vm.pmap.shpgperproc="1024" vfs.zfs.arc_min="256M" vfs.zfs.arc_max="384M" vfs.zfs.vdev.cache.size="50M" vfs.zfs.prefetch_disable="1" kern.maxproc="20000" zen# and zen# zpool status pool: datapool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM datapool ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ad24 ONLINE 0 0 0 ad8 ONLINE 0 0 0 ad18 ONLINE 0 0 0 ad20 ONLINE 0 0 0 ad22 ONLINE 0 0 0 ad10 ONLINE 0 0 0 spares ad26 AVAIL errors: No known data errors zen# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT datapool 5.44T 3.54T 1.90T 65% ONLINE - zen# Problem starts as soon as we put some load on FS - e.g. last night I tried to rsync 1TB of different files from above ZFS pool to single hard drive (partitioned UFS2), but server crashed and restarted after copying ~700GB of data. I also tried to stress-test it by running 100 torrent downloads (using rtorrent), and server also crashes and restarts after running for about 30-40 minutes. My guess these crashes happen due to big load on filesystem - ZFS eats all available memory and then server simply crashes. Right now I'm completely lost - I can't even copy 1TB from ZFS to another partition.. How can I diagnose the issue? Is there anything available to make ZFS more stable? Thanks, Mikhail.