From owner-freebsd-fs@FreeBSD.ORG Mon Oct 22 01:36:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 30030F59 for ; Mon, 22 Oct 2012 01:36:24 +0000 (UTC) (envelope-from jurgen.weber@theiconic.com.au) Received: from exprod6og110.obsmtp.com (exprod6og110.obsmtp.com [64.18.1.25]) by mx1.freebsd.org (Postfix) with SMTP id 9444D8FC0C for ; Mon, 22 Oct 2012 01:36:23 +0000 (UTC) Received: from mail-pb0-f72.google.com ([209.85.160.72]) (using TLSv1) by exprod6ob110.postini.com ([64.18.5.12]) with SMTP ID DSNKUISjEXtrkqaSN+PHtb4nIKPPaGfSDc5b@postini.com; Sun, 21 Oct 2012 18:36:23 PDT Received: by mail-pb0-f72.google.com with SMTP id rp2so6258148pbb.7 for ; Sun, 21 Oct 2012 18:36:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:x-gm-message-state; bh=1j3AjdbiAfBoTsp2vkctHPuBPwhrfsaQ+CidYM1kuoY=; b=VxvKKrdDW/ij7YPQbTNWH0XsnppS+dYDiisrUmkMU6u8Nb2Ymlr4RWiIc6pz8RKP6U p2EIWYEQ3vCAZH8If4HOPyyuJFAZpaoAbsTzc34i3van5dU0257GK7eDyOZr+k2V24yw wzrXp7uxnNM3hsmxLEjK4WOyJ7jRrbtIcPAURwhYMP+ALnsN++rnKmqwIYVQf1D4Z2as OYVG8U2/dv0KJpk3sUWP/xEwS07Lzf+X14ecslnRoZloG4TYWxZw+M6sBKrN9AK0mp4g 5hQP0OsHRe5/8r369nm2P0wHm43TB5IPU8jXjzcBKX4baPAa9lcHsM3CstYsXRkykg9g 9p+w== Received: by 10.68.224.161 with SMTP id rd1mr26150251pbc.49.1350869407360; Sun, 21 Oct 2012 18:30:07 -0700 (PDT) Received: by 10.68.224.161 with SMTP id rd1mr26150240pbc.49.1350869407246; Sun, 21 Oct 2012 18:30:07 -0700 (PDT) Received: from [172.20.24.157] ([202.126.107.170]) by mx.google.com with ESMTPS id c7sm2721858pay.10.2012.10.21.18.30.04 (version=SSLv3 cipher=OTHER); Sun, 21 Oct 2012 18:30:06 -0700 (PDT) Message-ID: <5084A19A.5050905@theiconic.com.au> Date: Mon, 22 Oct 2012 12:30:02 +1100 From: =?ISO-8859-1?Q?J=FCrgen_Weber?= User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: mfi0 timeout error zfs boot mount problem References: <508090E8.4010300@theiconic.com.au> <5081CE05.1010108@theiconic.com.au> <50830EA3.6020001@theiconic.com.au> <508471E0.9010805@theiconic.com.au> In-Reply-To: <508471E0.9010805@theiconic.com.au> X-Gm-Message-State: ALoCoQl7bzcxazGBWJfH6aXiyRV37KKIfetSfgODRimWnc2dqh0TQWLdLdwmHO7BzwxWMnuCe9jVogtAaLjznbzWlWhNZcQDbrLFw/MZDLeKJId0pV738bWTCtJhqFhIw4POdXAdPJqxwQgULubUOkNhjwqIIOpHIA== Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Oct 2012 01:36:24 -0000 Some more updates! on the bootloader I have also tried: kern.maxfiles=5000000 kern.maxvnodes=5000000 I have also gone into the Card settings BIOS and changed under advanced settings "Forward Read" to "none". Now the systems gets to "Trying to mount root from zfs:tank/root []..... " and then after maybe 1 to 5 minutes the next couple of lines load like its working! eg: "Setting hostuuid: xxxxx" "Setting hostid: xxxxxx" "Entropy harvesting:interrupts ethernet point_to_point kickstart" "Starting file system checks:" "Mounting local file systems:." and stops. I have had the machine on my desk all morning observing it and I can see the disk access is going crazy,, it is doing something. I have found this article: http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe I have a 15TB file system which has dedup on from the start (10TB. I feel its trying to load the DDT and its going to swap/there is not enough RAM (only have 16GB's). Hopefully my 64GB RAM upgrade is enough. Thanks Jurgen On 22/10/12 09:06, Jürgen Weber wrote: > This is still a problem for me, is anyone there? :) > > I have tried the following at the bootime loader. > > vfs.zfs.zil_disable="1" > vfs.zfs.prefetch_disable="1" > vfs.zfs.txg.timeout="5" > > Any other suggestions on how to get this zpool to import and mount again? > > Thanks > > On 21/10/12 07:50, Jurgen Weber wrote: >> Hi >> >> Lastly, is there a way at boot time, some sysctl's or something I can >> set to bring zfs to a minimalistic state? Turn off features, etc to >> get this to mount? >> >> Any ideas appreciated. >> >> Thanks >> >> Jurgen >> On 20/10/2012 9:02 AM, Jurgen Weber wrote: >>> Guys >>> >>> Some more details on this, some insight would be greatly appreciated. >>> >>> As my day wore on trying to get this zpool to import or mount I have >>> learnt a few things. I think over time this issue has came about as >>> more and more data was added to the file systems. >>> >>> Some further details: >>> >>> Its a 8 disk raidz pool that the system boots from as well. The disk >>> are all 2TB. >>> The server has 16GB Of RAM, I notcied the day before this happen the >>> server was struggling with its RAM griding to a halt and dumping its >>> RAM. >>> The issue is not hardware because I found another server (same one) >>> swapped the harddrives out took another 8GB of RAM and I have the >>> same problem. >>> The main data file systems have dedup and gzip compression on. >>> >>> I have booted from open/Oracle Solars 11 adn attempted to import and >>> the Solaris live CD will not import either. In the Solaris system >>> the disk detach from the system. >>> >>> I get the feeling that ZFS is hitting some root limit when >>> attempting to mount and its not finishing the job. >>> >>> Thanks >>> >>> Jurgen >>> >>> On 19/10/2012 10:29 AM, Jürgen Weber wrote: >>>> Team >>>> >>>> I have googled around for a solution and I see a lot of posts about >>>> firmware versions and patches for FreeBSD 8.*. >>>> >>>> I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has >>>> been running for months. >>>> >>>> Now it will not boot, I get the following: >>>> >>>> "Trying to mount root from zfs:tank/root []..... >>>> mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS >>>> (this just repeats). >>>> >>>> I have not seen this error before during normal runtime, _only_ >>>> during boot. >>>> >>>> Originally when I had the problem I could boot off a USB stick >>>> (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work >>>> on the livecd. Rebooting and the main system would work. >>>> >>>> This time this work around does not work for me. When I am on the >>>> USB stick I can run a 'zpool import' and all of the disk are >>>> recognised, the pool is recognised and the file system is healthy. >>>> >>>> The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515. >>>> Running FreeBSD 9.1-RC1, latest zfs and zpool versions. >>>> >>>> I have tried disabling the cache (mfiutil cache xxx disable). I >>>> have also gone into the Card settings and changed under advanced >>>> settings "adaptive forward read" to "read only". >>>> >>>> Any help, appreciated. >>>> >>>> Thanks >>>> >>> >> > -- Jürgen Weber Systems Engineer IT Infrastructure Team Leader THE ICONIC | E jurgen.weber@theiconic.com.au | www.theiconic.com.au