From owner-freebsd-fs@FreeBSD.ORG Thu Jun 27 15:07:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D4D89AF2 for ; Thu, 27 Jun 2013 15:07:51 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from mail-vb0-x233.google.com (mail-vb0-x233.google.com [IPv6:2607:f8b0:400c:c02::233]) by mx1.freebsd.org (Postfix) with ESMTP id 9BBB31C66 for ; Thu, 27 Jun 2013 15:07:51 +0000 (UTC) Received: by mail-vb0-f51.google.com with SMTP id x17so740727vbf.38 for ; Thu, 27 Jun 2013 08:07:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=53ttVKODMrGZ0L4KjM0snmloNb3+Lhik3WqPa7cQWZU=; b=MaM3CgebXZTAMZMfJIERSsOB5knB2Af5UG+rLbwRN5iqyna1TaFJJ28iyAjU6lYuS8 XPV3UvB8hXxYPsmUCu4CD/dqnRr6m6z4oZP9xhOnPhgtuwMCFA/e4jsndY+tKp8qiX/f svMf5vVEnbf+524eF4m43MidAMfu1QU0ldPRNgr+U8L8Eh93LiuFXlUP4691JLliAfjt r7Q8AdP0WQI8/q3UoxFnA58lYibNnPCFH18TKsfgl4wfJR+Wn1Lqi4aYDHfDiFz65mQF IFrFBexnvuJSxuegPgKu+lOgZoYv5yRwygDiFcN9fGQ3PujY1nPzq7UWoIpr2DL52+Ta 1pjQ== MIME-Version: 1.0 X-Received: by 10.220.124.1 with SMTP id s1mr3604154vcr.93.1372345671013; Thu, 27 Jun 2013 08:07:51 -0700 (PDT) Received: by 10.58.77.34 with HTTP; Thu, 27 Jun 2013 08:07:50 -0700 (PDT) Date: Thu, 27 Jun 2013 12:07:50 -0300 Message-ID: Subject: ZFS mount volume freeze From: Alexandre Biancalana To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jun 2013 15:07:51 -0000 Hi list, I'm using FreeBSD-8.3-RELEASE-p7 amd64, actually this is a FreeNAS 8.3.1-RELEASE but everything described here were also tested booting machine with a usb memstick using FreeBSD-10.0-CURRENT-amd64-20130623-r252101-memstick image with the same results. I've a home file server where I store my personal files, backups and virtual machine images served through nfs to a XCP. Yesterday my VMs stopped and all commands (ls, find) issued to the VMs volume (through NFS or local on fileserver) started to freeze hanging forever. Then I reboot the machine and it starts to hang after zpool import and before volumes get mounted. This machine has 8GB RAM and AMD Bulldozer six-core CPU. I have two zpool: $ zpool status pool: datastore0 state: ONLINE scan: resilvered 834G in 41h27m with 0 errors on Fri Jun 14 15:59:54 2013 config: NAME STATE READ WRITE CKSUM datastore0 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/d761e927-d2e2-11e2-bddc-902b34f47b09 ONLINE 0 0 0 gptid/daa3440d-d3c8-11e2-8405-902b34f47b09 ONLINE 0 0 0 errors: No known data errors pool: datastore1 state: ONLINE scan: scrub repaired 1.63G in 1h47m with 0 errors on Thu Jun 27 01:06:59 2013 config: NAME STATE READ WRITE CKSUM datastore1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada4 ONLINE 0 0 0 logs gptid/74f920fd-d301-11e2-8405-902b34f47b09 ONLINE 0 0 0 errors: No known data errors $ zfs list NAME USED AVAIL REFER MOUNTPOINT datastore0 1.21T 33.7G 2.12M /mnt/datastore0 datastore0/ale 534G 33.7G 9.10G /mnt/datastore0/ale datastore0/ale/backup 525G 33.7G 143G /mnt/datastore0/ale/backup datastore0/forensics 42.1G 33.7G 42.1G /mnt/datastore0/forensics datastore0/jail_plugins 1.54G 593M 1.42G /mnt/datastore0/jail_plugins datastore0/jail_plugins_archive 278M 746M 278M /mnt/datastore0/jail_plugins_archive datastore0/mirror 520G 33.7G 387G /mnt/datastore0/mirror datastore1 291G 171G 2.29M /datastore1 datastore1/iso 51.1G 28.9G 51.1G /tmp/iso datastore1/vms 238G 144G 55.6G /datastore1/vms datastore0 => mirror 2 1TB SATA datastore1 => mirror 2 500G SATA So I boot in single mode and can successfully import both pools, datastore0 is working perfectly I can mount all their volumes and access all data without any problem. The problem is with a specify volume where I store the VM files is located in datastore1/vms, when I try to mount this volume, no matter if I try todo zfs mount -a or zfs mount datastore1/vms the command hangs forever. I've another volume in that zpool (datastore1/iso) that I can mount and use without any problem. I already tried to do a scrub, that corrected some data but even after that I can mount that volume. Now I'm with the machine online with all volume mounted but datastore1/vms and waiting a zdb datastore1 to finish, (I can provide the cmd output too). How this machine is my home file server I'm free to do almost everything to troubleshoot that. Any ideas on how troubleshoot that further ? Regards, Alexandre